Mobile secretary cloud application

ABSTRACT

The invention provides a method, system, and a software program product for assisting a user and/or managing tasks of the user, by a mobile secretary cloud application configured to operate in a mobile client device and cloud server network. The mobile secretary cloud application reads data from another software application and operates at least one of another application and a third application based on the read data. Further, Artificial intelligence is utilized by the mobile secretary cloud application for operating another application and the third application.

TECHNICAL FIELD

The invention relates to managing tasks of a user through a smartphoneand more particularly to utilizing artificial intelligence in managingtasks of the user.

BACKGROUND

Modern smartphones carry numerous software applications for performingdifferent specialized tasks. For example, a calendar application keepsrecord of activities scheduled for particular dates, an e-mailingapplication manages e-mails received and sent by a user, and imagegallery manages images and videos captured using the smartphone.Similarly, there are numerous other applications for managing differentdata relevant to the user.

Many times, it becomes a tedious task for the user to manage suchnumerous applications. For example, the user may need to refer to afirst application and may need to manage a second application based onthe first application. For example, the user may need to access theemailing application for accessing an email comprising details of ameeting scheduled for the user. Based on retrieved details of themeeting, the user may need to revert with an acceptance of the meeting,set an alarm or a reminder for the date of meeting, set a navigationroute for reaching a venue of meeting, or book flight/train/bus forreaching the venue of the meeting. Similarly, the user may need tomanage several activities using different applications installed on hissmartphone, based on information retrieved from one application. Presentstate of art merely comprises management of tasks based on user'sinstructions, either provided manually or via speech.

For example, Siri software in the iPhone may take verbal instructionsfrom a user to operate the iPhone.

For example, US20130110519 describes a method to operate an intelligentdigital assistant for assisting a user in task management. Theintelligent digital assistant stores a plurality of predefined domainsindicative of areas of services offered. From a user's request (speechinput), one or more words are derived. Thereafter, a match is identifiedbetween the words derived from the user's request and words associatedwith the plurality of predefined domains. Based on a degree of match, arelevant domain for the user request is identified. Successively, a taskis executed based on steps specified in a task flow associated with therelevant domain and task parameters derived from the user's request.

WO2014197635A2 describes an intelligent automated assistant system. Theintelligent automated assistant system receives a search request from auser. The search request includes at least a speech input and one ormore search criteria for identifying reservable items. In response toreceiving the search request, a plurality of search results,corresponding to reservable items, are presented to the user.Successively, a reservation request is received for one search resultamongst the plurality of search results. In response to the reservationrequest, a reservation procedure is executed to reserve a respectivereservable item.

US20160155442 describes a digital personal assistant configured toperform tasks based on user instructions. Firstly, a user input isreceived, in form of a voice command or a text, for performing a task.Speech recognition is performed on the user input provided as voicecommand. Thereafter, the task is recognized and a registered actionprovider to provide with the task details. Successively, instructionsare sent to the particular action provider for performing the task. Userpreferences are collected, and for example if the user selects oneaction provider 5 times in a row, on the 6^(th) time that provider isset as the default option.

Thus, the prior art features a number of simplistic workflows andautomations, and even discloses very simple steps for learning thepreference of the user in terms of one criterion. Individual automationsof tasks, however, are not capable of reducing the wholisticadministrative burden of the smartphone user in a significant way.

SUMMARY

It is an object of the invention to address and improve aforementioneddeficiencies in above discussed prior art(s). The object of theinvention is to provide an electronic artificial intelligencesoftware-based secretary to every user, at a minimal fraction of thecost that it costs to hire a human secretary.

The central philosophy of the invention is not to guess what the userwould do, given a certain input into the device. Ordinary consumers arenotoriously inefficient in managing their time. Successful people becomesuccessful partly because they have hired and employ successful personalassistants (PA's) and secretaries. The central philosophy of theinvention is how a professional secretary would handle the input, tomaximize the benefit to the user. In order to achieve this aim,technical software embodiments are produced in the following.

It is an object of the invention to assist a user and/or manageadministrative tasks of the user. Such assistance is provided to theuser by a mobile secretary cloud application configured to operate in amobile client device and cloud server network. Present invention isdescribed using systems, methods, and software program products, asexplained below.

A method for assisting a user and/or managing tasks of the user, by amobile secretary cloud application configured to operate in a mobileclient device and cloud server network, is in accordance with theinvention and characterized by the following steps,

-   -   the mobile client device is a smartphone connected to at least        one cloud server network via a wireless communication network,    -   a data analyzing module of the mobile secretary cloud        application reads data from at least one other application,    -   a data modifying module of the mobile secretary cloud        application operates the at least one other application, based        on data read from the at least one other application, and    -   an artificial intelligence module of the mobile secretary cloud        application uses artificial intelligence to operate the at least        one other application for assisting a user or to use the at        least one other application independently for performing at        least one task for the user.

The mobile secretary cloud application may replicate secretaryfunctions. In one case, the at least one other application is an e-mailor messaging application. The mobile secretary cloud applicationanalyses messages and automatically sorts the messages to differentfolders. Further, the mobile secretary cloud application analysesmessages and selects an indication sound automatically based on contentsof the messages and sender and/or recipients of the messages. In anothercase, the at least one other application is a telephony application, andthe mobile secretary cloud application makes phone calls on behalf ofthe user, automatically answers to messages, and prompts queries onbehalf of the user, using a synthesized voice on a phone line. In yetanother case, the at least one other application is a map application,and the mobile secretary cloud application provides a route calculatorthat provides a combined route of driving and walking based on e-mails,calendar entries, and location search. In a preferred embodiment thesecretary application uses artificial intelligence to determine whatwould be the optimum way of conducting the three activitiesindividually. Then, further in a preferred embodiment, the secretaryapplication performs an optimum total fit of all of the combinedactivities for the user's day. This may involve for example doing theautomated phone calls during the lunch break of the user, as an eatinguser is less likely to talk. The secretary application will propose adraft email response to a received email, selecting a short matter witha short response for a 10-minute break in the schedule of the user. Incontrast for an entire silent afternoon in the office, the secretaryapplication will present a 30-page project plan for inspection, as thesecretary application knows that now is the convenient time for that.Similarly, for travel, the shortest route that allows the execution ofthe most urgent matters is selected in accordance with the invention.

A method for assisting a user and/or managing tasks of the user, by amobile secretary cloud application configured to operate in a mobileclient device and cloud server network is in accordance with theinvention and characterized by the following steps,

-   -   the mobile client device is a smartphone connected to at least        one cloud server network via a wireless communication network,    -   a data analyzing module of the mobile secretary cloud        application is configured to read data from at least one other        application,    -   a data modifying module of the mobile secretary cloud        application is configured to operate a third application,        different from the at least one other application, based on data        read from the at least one other application,    -   an artificial intelligence module of the mobile secretary cloud        application uses artificial intelligence to operate the third        application for assisting a user or to use the third application        independently for performing at least one task for the user.

In one embodiment, the third application is an e-mail or messagingapplication. The mobile secretary cloud application analyses messagesand automatically sorts messages to different folders based on dataderived from a calendar application or an Internet browser. Further, themobile secretary cloud application analyses messages and selects anindication sound automatically based on contents of the messages andsender and/or recipients of the messages, using a text analyzerapplication as the at least one other application.

In another embodiment, the third application is a telephony application,and the mobile secretary cloud application makes phone calls on behalfof the user, automatically answers to messages on behalf of the user,and prompts queries to the user. The mobile secretary cloud applicationperforms such functions using synthesized voice on the phone line foroutgoing voice and using a voice recognition software application forincoming voice as the at least one other application.

In yet another embodiment, third application is a map application, andthe mobile secretary cloud application provides a route calculator forproviding a combined route of driving and walking based on data from theat least one other application, which data may comprise e-mails,calendar entries, search application, and Internet browser.

A mobile client device configured to execute a mobile secretary cloudapplication, wherein the mobile client device is a smartphone connectedto at least one cloud server network via a wireless communicationnetwork, and the mobile client device comprises:

-   -   a processor; and    -   a memory connected to the processor, wherein the processor is        configured to execute the mobile secretary cloud application to:        -   read data, using a data analyzing module, from at least one            other application;        -   operate, using a data modifying module, the at least one            other application based on data read from the at least one            other application; and        -   operate, using an artificial intelligence module, the at            least one other application for assisting a user or to use            the at least one other application independently for            performing at least one task for the user.

A mobile client device configured to execute a mobile secretary cloudapplication, wherein the mobile client device is a smartphone connectedto at least one cloud server network via a wireless communicationnetwork, and the mobile client device comprises:

-   -   a processor; and    -   a memory connected to the processor, wherein the processor is        configured to execute the mobile secretary cloud application to:        -   read data, using a data analyzing module, from at least one            other application;        -   operate, using a data modifying module, a third application,            different from the at least one other application, based on            data read from the at least one other application,        -   operate, using an artificial intelligence module, the third            application for assisting a user or to use the third            application independently for performing at least one task            for the user.

A software program product stored in a memory medium for assisting auser and/or managing tasks of the user, by a mobile secretary cloudapplication, characterized by the following steps,

-   -   the software program product reads data from at least one other        application;    -   the software program product operates the at least one other        application based on data read from the at least one other        application; and    -   the software program product utilizes artificial intelligence to        operate the at least one other application for assisting the        user or to use the at least one other application independently        for performing at least one task for the user.

A software program product stored in a memory medium for assisting auser and/or managing tasks of the user, by a mobile secretary cloudapplication, characterized by the following steps,

-   -   the software program product reads data from at least one other        application;    -   the software program product operates a third application,        different from the at least one other application, based on data        read from the at least one other application; and    -   the software program product utilizes artificial intelligence to        operate the third application for assisting a user or to use the        third application independently for performing at least one task        for the user.

The best mode of the invention is considered to be a mobile secretaryapplication that is connected to the cloud servers. In the best mode,the cloud network continuously collects and updates self-learning filesfor a number of secretarial tasks, and makes them available to themobile secretary applications on the mobile client terminals. In thebest mode, the mobile secretary application and the cloud network firstdetermine what are the correct draft responses to each incoming taskindividually. Once the individual responses or actions to tasks havebeen determined, the agenda with which these responses or actions aremost efficiently completed with the user in a unit of time, for examplea day or a week, is determined. Once the agenda is determined, thesecretary application and cloud network guide the user through theresponses and activities in accordance with the determined agenda. Somesecretarial activities are also operated on a continuous and instantbasis, for example the playing of indication sounds for arrivingmessages, in the best mode.

The best mode can be applied to a number of uses, some of which mayoperate independently of the cloud network for long periods. Forexample, in the best mode, the selection of an indication sound isautomatically based on contents of the messages and sender and/orrecipients of the messages. In one case, AI is utilized to processcontents of the messages and thereby select an indication sound based oncontents of the messages. For example, the mobile client device of theuser may receive a pleasant message, stating “You did an incrediblejob.” Similarly, the user may receive other pleasant messages, such as“Thank you for your kind support,” “I am glad to hear about yourpromotion,” and “love you my son.” The email and message processingmodule 510, utilizing AI, may learn upon several such messages, and maydetermine that such messages convey pleasant gestures. The email andmessage processing module 510 may learn based on processing of wordspresent in the emails and messages. For example, in present case, thewords “incredible,” “thank you,” “kind,” “glad,” “promotion,” and “love”infer pleasant gestures from a sender, and the email and messageprocessing module 510 may learn such meaning. Based on such learning,the email and message processing module 510 may play uplifting orromantic audio, respectively, as the indication sound.

In contrast, when the mobile client device of the user may receive aharsh message, stating “poor job.” Similarly, the user may receive otherharsh messages, such as “you've passed the deadline,” “I won't toleratesuch careless attitude,” and “hate you.” The email and messageprocessing module 510, utilizing AI, may learn upon several suchmessages, and may determine that such messages convey harsh gestures.The email and message processing module 510 may learn based onprocessing of words present in the emails and messages. For example, inpresent case, the words “poor,” “passed deadline,” “won't tolerate,” and“hate” infer harsh gestures from a sender, and the email and messageprocessing module 510 may learn such meaning. Based on such learning,the email and message processing module 510 may play a loud beep orwarning sound as the indication sound.

Further, the email and message processing module 510 may also derivemeanings based on analysis of smileys/emoticons and special characterspresent in the emails and messages. The special characters may include“!!,” “?” and other known special characters. For example, the user mayreceive a message stating “you again!!” or “what have you done?” Theemail and message processing module 510 may learn that harsh gesturesare implied by such messages. Based on such learning, the email andmessage processing module 510 may play the loud beep or warning sound asthe indication sound. The indication sound selection embodiment can beimplemented as a cloud server—client device configuration, but also inthe client device only configuration as it requires few or no updatesfrom the network. In an alternative implementation of the best mode, thesubject line of the email or message or some content of the message isread out loud to the user, so that the user gets a quick indication ofthe contents without having to access the message itself via the screen.

A method of providing an alert to a mobile device user, comprising amobile device and at least one cloud server and a communication network,is in accordance with the invention and characterized by the steps of,

-   -   the mobile device and/or cloud server store appointment data        and/or a calendar entry data with a time and a location or a        variable location or undefined location,    -   mobile device of user and/or cloud server calculate the travel        time to the appointment location from the user location using        map and/or navigation software available over Internet or cloud        network or on mobile device,    -   mobile device of user and/or cloud server configures a sound,        vibration and/or visual alert and/or the said alert for a        calendar entry, and/or a calendar entry,    -   the time of the said alert and/or calendar entry is at a time        that precedes the appointment by the travel time, thereby        allowing the user to depart his location at the correct time,        and thereby allowing the user to arrive at the appointment at        the correct time after having travelled to the appointment.

In one aspect of the invention the location of the appointment ismeasured from a location of another mobile device of an appointmentparty other than the user, disclosed to the user mobile device via thecloud server network, deduced from the Internet by location search, orprovided by messaging to the mobile device of user. In this case theappointment location, the travel time and the alert time may becontinuously updated as location of mobile device of user, location ofthe other appointment party or location of his or her mobile device,route between user and appointment location, traffic and other factorsthat would influence the travel time change, and/or are anticipated tochange by user mobile device and/or cloud server software. This softwareis typically the mobile cloud secretary application as explained in thisapplication. The mobile cloud secretary application or user mobiledevice and/or cloud server software may also forecast traffic conditionsand route times, using artificial intelligence or otherwise as explainedin this application. This can be done by using daily traffic patternsstored in historical data and calculating expected traffic conditionsand/or route times for a future time period. The route time ispreferably for a route that leads from the user location to theappointment during the time preceding the appointment, so that the userarrives at the appointment location, not too early and not too late, buton time.

A telecommunication system for providing an alert to a mobile deviceuser, comprising a mobile device and at least one cloud server and acommunication network, is in accordance with the invention andcharacterized in that,

-   -   the mobile device and/or cloud server are configured to store        appointment data and/or a calendar entry data, optionally into a        calendar application, with a time and a location or a variable        location or undefined location,    -   mobile device of user and/or cloud server is configured to        calculate the travel time from user location to the appointment        location using map and/or navigation software available over        Internet or cloud network,    -   mobile device of user and/or cloud server configures a sound,        vibration and/or visual alert and/or the said alert for a        calendar entry, and/or a calendar entry,    -   the time of the said alert and/or calendar entry is configured        to a time that precedes the appointment by the travel time,        thereby allowing the user to depart his location at the correct        time, and thereby allowing the user to arrive at the appointment        at the correct appointment time after having travelled to the        appointment.

In one aspect of the invention the location of the appointment isconfigured to be measured from a location of another mobile device of anappointment party other than the user disclosed to the mobile device ofthe user via the cloud server network, and/or deduced from the Internetby location search, and/or provided by messaging to the mobile device ofuser. In this case the appointment location, the travel time and thealert time may be continuously updated by the mobile device and/or cloudserver as location, route, traffic and other factors that wouldinfluence the travel time change, and/or are anticipated to change bythe mobile device and/or cloud server. The mobile device and/or cloudserver may also forecast traffic conditions and route times, usingartificial intelligence or otherwise as explained in this application.This can be done using daily traffic patterns stored in historical dataand calculating expected traffic conditions and/or route times at thetime and place of the appointment in the future. This functionality ispreferably realized with the mobile cloud secretary applicationdescribed in this application.

A software program product stored in a memory medium for providing analert to a mobile device user, configured to operate in a mobile deviceand at least one cloud server via a communication network, is inaccordance with the invention and characterized in that,

-   -   the mobile device and/or cloud server software is configured to        store appointment data and/or calendar entry data, optionally        into a calendar application, with a time and a location or a        variable location or undefined location,    -   mobile device of user and/or cloud server is configured to        calculate the travel time to the appointment location from a        user location using map and/or navigation software available        over Internet or cloud network,    -   mobile device of user and/or cloud server configures a sound,        vibration and/or visual alert and/or the said alert for a        calendar entry, and/or a calendar entry,    -   the time of the said alert and/or calendar entry is configured        to a time that precedes the appointment by the travel time,        thereby allowing the user to depart his location at the correct        time, and thereby allowing the user to arrive at the appointment        at the correct appointment time after having travelled to the        appointment.

In one aspect of the invention, the location of the appointment isconfigured to be measured from a location of another mobile device of anappointment party other than the user. This location is disclosed to theuser mobile device via the cloud server network, deduced from theInternet by location search, or provided by messaging to the mobiledevice of user. In this case the appointment location, the travel timeand the alert time may be continuously or irregularly updated by theuser mobile device and/or cloud server software as location, route,traffic and other factors that would influence the travel time change,and/or are anticipated to change by mobile device and/or cloud serversoftware. This cloud software, for example embodied as mobile cloudsecretary application software as explained in this application inmultiple embodiments, may also forecast traffic conditions and routetimes, using artificial intelligence or otherwise as explained in thisapplication. This can be done using daily traffic patterns stored inhistorical data and calculating expected traffic conditions and/or routetimes at the appointment time and to the appointment place in thefuture.

Needless to say, approximate values in the proximity of the appointmenttime and place can be used in accordance with the invention to makecalculations. Needless to say, as can be seen the invention caters for avariety of users, including those people who a) use only an alert at atime preceding the appointment, b) use only a calendar entry at a timepreceding the appointment, c) use a calendar entry at the appointmenttime with also an alert at a time preceding the appointment time.

It should be noted that this invention has the great advantage ofallowing the appointment party other than the user to move with his orher mobile device. The user can just home in on his or her location,using the invention, and meet him or her there. The invention thereforeelimates a lot of the wasted time in being too early, or being late, orspending time trying to find the other person.

The aforementioned invention can be realized as a mobile cloud secretaryapplication software or as a feature of the operating system and utilityapps that come with the operating system. This applies to allembodiments of the invention in this application.

Alternatively, the invention can be realised as any mix of a 3^(rd)party application, operating system features and/or applications thatare installed with the operating system, and/or cloud service softwareto which the mobile device of user subscribes to. This applies to allembodiments of the invention in this application.

The “cognitive process”, “self-improved activity” are terms of machinelearning that have been defined in “Introducing Artificial IntelligenceA Graphic Guide, Henry Brighton, Iconbooks 2003” and “Machine LearningThe New AI, Ethem Alpaydin, The MIT Press 2016”. These books are citedhere as reference. The book “The Technological Singularity, MurrayShanahan, The MIT Press, 2015.” is also cited here as reference.

A method of providing automatic secretarial support for a mobile phoneuser comprises a mobile client device and cloud servers and cloudsoftware and is characterized by the following steps:

-   -   a mobile cloud secretary application records user inputs thereby        recording the cognitive process of the user,    -   the mobile secretary cloud application repeats new instructed        activities as recorded by the cognitive process of the user,    -   the mobile secretary application derives a self-improved        activity from crowd sourced data,    -   the mobile secretary application replaces an instructed activity        with a self-improved activity.

In one aspect of the invention the mobile secretary application recordswhether the user accepts the self-improved activity, corrects theself-improved activity or rejects the self-improved activity with hisown activity.

In some aspects of the invention the mobile secretary applicationrecords an accepted self-improved activity as a default option forfuture use. In another aspect, the mobile secretary application recordsthe user corrected version of the corrected self-improved activity as adefault option for future use. In a further aspect of the invention, themobile secretary application records the rejected self-improved activityas failed and does not suggest it again.

A telecommunication system comprising mobile client devices and cloudserver computers, and a mobile secretary cloud software is in accordancewith the invention and characterized in that, the mobile secretary cloudsoftware either comprises or is configured to operate:

-   -   a cognitive software module configured to record user inputs and        derive and record the cognitive process of the user,    -   a self-improvement module configured to analyze the cognitive        process of the user and suggest improvements by replacement to        the machine learned cognitive processes,    -   a crowd-sourcing module from which the self-improvements to        machine learned processes are configured to be derived.

A mobile secretary cloud application software program product stored ina memory medium, characterized in that it either comprises or isconfigured to operate:

-   -   a cognitive software module configured to record user inputs and        derive and record the cognitive process of the user,    -   a self-improvement module configured to analyze the cognitive        process of the user and suggest improvements by replacement to        the machine learned cognitive processes,    -   a crowd-sourcing module from which the self-improvements to        machine learned processes are configured to be derived.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an embodiment 10 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatinganother software application, according to an embodiment.

FIG. 1B illustrates an embodiment 11 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operating athird software application, using another software application,according to an embodiment.

FIG. 2A illustrates an embodiment 20 of a block diagram of a system 200for executing a mobile secretary cloud application, according to anembodiment.

FIG. 2B illustrates a machine learning embodiment 21 where artificialintelligence is used to distinguish legitimate business emails from spamemails and other emails.

FIG. 3A illustrates an embodiment 30 of a user interface showing amobile secretary cloud application operating another softwareapplication, according to an embodiment.

FIG. 3B illustrates an embodiment 31 of a user interface showing amobile secretary cloud application utilizing another softwareapplication for operating a third software application, according to anembodiment.

FIG. 4A illustrates an embodiment 40 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral other software applications, according to an embodiment.

FIG. 4B illustrates an embodiment 41 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral third software applications, using another software application,according to an embodiment.

FIG. 5 illustrates an embodiment 50 of a block diagram of a systemexecuting a mobile secretary cloud application.

FIG. 6A illustrates an embodiment 60 of a user interface showing amobile secretary cloud application operating an e-mail/messagingapplication.

FIG. 6B illustrates an embodiment 61 of a user interface showing amobile secretary cloud application operating an e-mail/messagingapplication.

FIG. 6C illustrates an embodiment 62 of a user interface showing amobile secretary cloud application operating a telephony application.

FIG. 6D illustrates an embodiment 63 of a user interface showing amobile secretary cloud application operating a map application.

FIG. 7 demonstrates a machine-learning embodiment 71 of the invention asa flow diagram.

FIG. 8A shows an embodiment 81 of the invention wherein the machinelearning modules are separate from the mobile secretary cloudapplication.

FIG. 8B shows an embodiment 82 of the invention where the machinelearning modules are integrated into the mobile secretary cloudapplication.

Some embodiments of the invention are described in the dependent claims.

DETAILED DESCRIPTION

The present disclosure provides a mobile client device, a method, and asoftware program product for assisting a user and/or managing tasks ofthe user. The tasks may be managed by a mobile secretary cloudapplication accessing an application installed in a mobile client deviceof the user. The mobile secretary cloud application may be availableover Google Play™ store for downloading over Android smartphones andover App Store™ for downloading over iOS™ smartphones. The mobilesecretary cloud application may either be available for free or as apaid application.

The mobile secretary cloud application is henceforth explained toutilize Artificial Intelligence for several purposes. Known machinelearning tools/deep learning frameworks may be utilized with or withoutmodifications. A few such known machine learning tools comprise Caffe™,Api.ai™, TensorFlow™, Mahout™, OpenNN™, H20™, MLlib™, NuPIC™, OpenCyc™,Oryx 2™, PredictionIO™, SystemML™, TensorFlow™, and Torch™.

FIG. 1A demonstrates an embodiment 10 of a method of implementation of amobile secretary cloud application for operating another softwareapplication, as a flow diagram. The method could be implemented in asystem identical or similar to embodiment 20 and 50 in FIG. 2 and FIG. 5, for example. The end-user of the method could use a user interfaceidentical or similar to that disclosed with embodiment 30, 31, 60, 61,62, and 63 in FIG. 3A, FIG. 3B, FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D.

FIG. 2 illustrates an embodiment 20 of a block diagram of a clientsystem 200 executing the mobile secretary cloud application. The systemcomprises interface(s) 202, processor 204, Graphical Processing Unit(GPU) 206, and memory 208. The memory 208 comprises a data analyzingmodule 210, a data modifying module 212, and an Artificial Intelligence(AI) module 214. Different phases of FIG. 1A and FIG. 1B will now beexplained in conjunction with modules of FIG. 2 .

Interface(s) 202 are used to interact with or program the system 200.The interface(s) 202 may either be a Command Line Interface (CLI) or aGraphical User Interface (GUI) or both.

The processor 204 may refer to any one or more microprocessors, finitestate machines, computers, microcontrollers, digital signal processors,logic, a logic device, an electronic circuit, an application specificintegrated circuit (ASIC), a field-programmable gate array (FPGA), achip, etc., or any combination thereof, capable of executing computerprograms or a series of commands, instructions, or state transitions.The processor 204 may also be implemented as a processor set comprising,for example, a general-purpose microprocessor and a math or graphicsco-processor. The processor 204 may be selected, for example, from theIntel® processors such as the Itanium® microprocessor or the Pentium®processors, Advanced Micro Devices (AMD®) processors such as the Athlon®processor, UltraSPARC® processors, microSPARC™ processors, HP®processors, International Business Machines (IBM®) processors such asthe PowerPC® microprocessor, the MIPS® reduced instruction set computer(RISC) processor of MIPS Technologies, Inc., RISC based computerprocessors of ARM Holdings, Motorola® processors, etc. The control unitdisclosed herein is not limited to employing the processor 204. Thecontrol unit may also employ a controller or a microcontroller and otherelectronics components.

The memory 208 includes a computer readable medium. A computer readablemedium may include volatile and/or non-volatile storage components, suchas optical, magnetic, organic or other memory or disc storage, which maybe integrated in whole or in part with a processor, such as processor204. Alternatively, all or part of the entire computer readable mediummay be remote from processor 204 and coupled to processor 204 byconnection mechanism and/or network cable. In addition to memory 208,there may be additional memories that may be coupled with the processor204 or the GPU (Graphics Processing Unit) 206.

In an embodiment, the system 200 is integrated with a cloud server 216,via a communication network 220. The cloud server may comprise a GPU 218or multiple GPUs. The communication network 220 used for thecommunication in the invention is the wireless or wireline Internet orthe telephony network, which is typically a cellular network such asUMTS- (Universal Mobile Telecommunication System), GSM- (Global Systemfor Mobile Telecommunications), GPRS- (General Packet Radio Service),CDMA- (Code Division Multiple Access), 3G-, 4G-, Wi-Fi and/or WCDMA(Wideband Code Division Multiple Access)-network.

In an example, the cloud server 216 may comprise a plurality of servers(not shown). In an example implementation, the cloud server 216 may beany type of a database server, a file server, a web server, anapplication server, etc., configured to store data related to the mobilesecretary cloud application and/or other applications. In anotherexample implementation, the cloud server 216 may comprise a plurality ofdatabases for storing the data files. The databases may be, for example,a structured query language (SQL) database, a NoSQL database such as theMicrosoft® SQL Server, the Oracle® servers, the MySQL® database, etc.The cloud server 216 may be deployed in a cloud environment managed by acloud storage service provider, and the databases may be configured ascloud-based databases implemented in the cloud environment.

Due to the self-learning or machine learning aspects of the inventionalso a neural network type of implementation for the client device 200and/or the cloud server 216 is possible. In some embodiments both theclient device 200 and/or the cloud server 216 may be realized as any ofthe following: SIMD (Single Instruction, Multiple Data) machine, MIMD(Multiple Instruction, Multiple Data) machine, and/or NIMD (NeuralInstruction, Multiple Data) machine.

The cloud server 216 which may include an input-output device usuallycomprises a monitor (display), a keyboard, a mouse and/or touch screen.However, typically there is more than one computer server in use at onetime, so some computers may only incorporate the computer itself, and noscreen and no keyboard. These types of computers are typically stored inserver farms, which are used to realize the cloud network used by thecloud server 216 of the invention. The cloud server 216 can be purchasedas a separate solution from known vendors such as Microsoft and Amazonand HP (Hewlett-Packard). The cloud server 216 typically runs Unix,Microsoft, iOS, Linux or any other known operating system, and comprisestypically a microprocessor, memory, and data storage means, such as SSDflash or Hard drives. To improve the responsiveness of the cloudarchitecture, the data is preferentially stored, either wholly orpartly, on SSD i.e. Flash storage. This component is eitherselected/configured from an existing cloud provider such as Microsoft orAmazon, or the existing cloud network operator such as Microsoft orAmazon is configured to store all data to a Flash based cloud storageoperator, such as Pure Storage, EMC, Nimble storage or the like. UsingFlash as the backbone storage for the cloud server 216 is preferreddespite its high cost due to the reduced latency that is required and/orpreferred for retrieving user data, user preferences, and data relatedto mobile/software applications etc.

The GPU (Graphical Processing Unit) 206 or 218 may refer to anelectronic circuit designed to manipulate and alter computer graphics,images, and memory to accelerate the analysis and creation ofimages/patterns. GPUs can be used in embedded systems, mobile phones,personal computers, workstations, game consoles, etc. The GPU 206 or 218may be selected, for example, from AMD GPUs, Nvidia GPUs, Intel GPUs,Intel GMA, Larrabee, Nvidia PureVideo, SoC, etc. In the invention,typically the machine learning or Artificial Intelligence parts of theprocessing are configured to be executed by the GPU, due to the largenumber of parallel processing or comparative processing required inmachine learning.

The system 200 may be configured as a mobile terminal computer,typically a smartphone and/or a tablet that is used to manage tasks ofthe user by operating software applications installed on the smartphone,etc. The system 200 is typically a mobile smartphone, such as iOS,Android or a Windows Phone smartphone. The GPU 206 or 506 present in thesmartphone or GPU 218 or 518 present in cloud server 216 or 516respectively, may process data of software applications installed on thesmartphone. The processed data may be used for managing the tasks of theuser. All of the three configurations are possible in accordance withthe invention, the first where the GPU 206 is only on the mobile clientdevice 200, the second where the GPU 218 is only on the cloud server216, the third where both the cloud server 218 and the mobile terminaldevice 200 have the GPU 206, 218.

However, it is also possible that the system 200 is a mobile station,mobile phone or a computer, such as a PC-computer, AppleMacintosh-computer, PDA-device (Personal Digital Assistant). The system200 could further be a device having an operating system such as any ofthe following: Microsoft Windows, Windows NT, Windows CE, Windows PocketPC, Windows Mobile, GEOS, Palm OS, Meego, Mac OS, iOS, Linux, BlackBerryOS, Google Android and/or Symbian or any other computer or smart phoneoperating system.

The description encompasses several modules, wherein the modules are tobe interpreted as programmed instructions stored in a segment of thememory 208 or 508, which when executed by the processor 204 or GPU 206,218 performs certain functionalities.

Any features of embodiment 20 may be readily combined or permuted withany of the other embodiments 10, 11, 21, 30, 31, 40, 41, 50, 60, 61, 62,63, 71, 81, and/or 82 in accordance with the invention.

In phase 102, the data analyzing module 210 of the mobile secretarycloud application or the GPU 206 and/or 218 reads data from anothersoftware application. Another software application indicates a mobileapplication different from the mobile secretary cloud application.Reading the data signifies extracting data entries related to a user.For example, when a calendar application is accessed, all eventsscheduled in a particular month, dates and timings of such events, andpriorities associated with such events are accessed. The events maycomprise meetings, interviews, journeys, birthdays, anniversaries, etc.In one case, the GPU 206 and/or 218 may read the data belonging toanother software application from Random Access Memory (RAM) or cachememory. Further, metadata belonging to the data may also be fetched inone embodiment.

In another embodiment, an e-mail application may be accessed to retrieveall e-mails present in a mailbox of the user. The data analyzing module210 of the mobile secretary cloud application or the GPU 206 and/or 218may access the e-mail application. All e-mails present in the e-mailboxi.e. received, sent, transmission in progress, and recently deleted maybe accessed.

In phase 104, the data modifying module 212 of the mobile secretarycloud application or the GPU 206 and/or 218 operates another softwareapplication based on data read from another software application. Forexample, upon accessing data from a messaging application, the datamodifying module 212 or the GPU 206 and/or 218 may manage messagespresent in the messaging application. In one case, the data modifyingmodule 212 or the GPU 206 and/or 218 may read details related to eachmessage present in the messaging application, such as details ofsenders, content of a message, and contents of other messages receivedfrom a sender. Based on processing of such details, the data modifyingmodule 212 or the GPU 206 and/or 218 may segregate each message into arespective category, such as segregation by category of message,segregation based on sender of messages, and segregation by priority ofmessages.

In another embodiment, upon accessing the e-mails, the data modifyingmodule 212 or the GPU 206 and/or 218 may manage the e-mails. The datamodifying module 212 or the GPU 206 and/or 218 may sort the e-mailsbased on priority, time, name of sender(s), name of receiver(s), andcontents of the e-mails. For example, while the user receives an e-mailtowards an acknowledgement of an online order, an instant pop-up messagemay be displayed on the system 200. Successively, based on processing ofcontent present in the e-mail, the data modifying module 212 or the GPU206 and/or 218 may mark the email as read, automatically archive, and/ordelete the e-mail.

In one embodiment, the GPU 206 and/or 218 may utilize ConvolutionalNeural Network (CNN) for processing the data related to the messagingapplication, e-mail application, or other applications. For example,there could be numerous messages present from several senders, and theGPU may perform multi-thread processing for obtaining precise and timelyoutput based on the processing. In one case, the GPU may compriseCombined Unified Device Architecture (CUDA) for enabling themulti-thread processing of the data. For example, the GPU 206, 218and/or processor 204 could generate draft response emails in response toreceived emails. To these emails the GPU 206, 218 and/or processor 204would automatically write text that they have machine learned to berelevant or deduce by Artificial Intelligence that will almost certainlybe present in the response email. For example, if there are multipleemails that address a certain party “Dear Sir/Madam”, the mobilesecretary cloud application will begin the draft response email in thatway too.

In phase 106, the Artificial Intelligence (AI) module 214 of the mobilesecretary cloud application or the GPU 206 and/or 218 utilizesArtificial Intelligence (AI) to operate another software application.The AI module 214 or the GPU 206 and/or 218 operates another softwareapplication for assisting a user and/or to perform a task for the user.In one case, to segregate the messages present in the messagingapplication, the AI module 214 or the GPU 206 and/or 218 utilizes AI forprocessing of different details related to each message. AI may be usedto process content of each message to segregate each message into arespective category. Processing the content of each message may compriseprocessing of title, words, hyperlinks, and dates present in eachmessage. Based on processing of such content, the AI module 214 or theGPU 206 and/or 218 utilizing AI may segregate each message into arelated category.

In addition to analysing the semantic content of messages with AI, themessages can be visually analysed using AI. Consider an example of usingconvolutional neural networks (CNN) for recognizing an email that justarrived in the inbox. CNN refers to artificial neural networks thatmodel visual perception by an animal or a human. The CNN algorithms maybe employed for image recognition tasks preferably emulating the waythat a human secretary views an email in accordance with the invention.The CNN comprises multiple layers of receptive fields that are smallneuron collections configured to process portions of an input image. Theoutput of each layer is successively tiled such that the input regionsoverlap to obtain a representation of the original image. In anembodiment, a deep learning framework called Caffe that uses C++,MATLAB, and Python programming languages is used for implementing theCNN. Caffe is a CNN library that is configured to support both CPU 290and GPU operations. In this example, the GPU 210 or 240 used by themobile secretary cloud application 200 may for example be a NVIDIA GPUwith 15 GB RAM.

A pre-defined dataset comprising, for example, 30000 images oflegitimate business emails and spam emails is used as a training datasetfor training the network. The training dataset comprises labelsassociated with each image. In an example, the training dataset isdownloaded from Kaggle which is a predictive modelling and analyticsplatform. The labelled images 422 and 424 are pre-processed and storedin a Python script format.

In this example, the GPU 206 and/or 218 executes histogram equalizationon the labelled images 422 and 424 of the training dataset. Histogramequalization is a technique used to adjust image intensities by usingthe image's histogram features. Histogram equalization enhances contrastof the images 422 and 424. The images resulting after histogramequalization of the images 422 and 424 are illustrated by images 426 and428, respectively. The GPU 206 and/or 218 performs image resizing toresize the images of the emails, for example, to a 227×227 format. Eachimage of email 430, 432 is labelled after performing the histogramequalization. The training dataset is then divided into 2 subsets. Firstsubset 434 called the training set comprises ⅚^(th) portion of thetraining images that are used for training a model. The second subsetcalled the validation set 436 comprises ⅙^(th) portion of the trainingimages of emails that are used for calculating and validating accuracyof the model. The subsets 434 and 436 are stored in the cloud databaseof the cloud server 216.

Features such as, histogram of oriented gradients (HoG), Scale-invariantfeature transform (SIFT), etc., of the images are extracted from thetraining images 430 and 432 by using a feature extraction software suchas, MATLAB. The extracted image features provide a description offeatures of an object present in an image 430 or 432 that are used inimage classification. Once the subsets 434 and 436 are created, the GPU206/218 generates the mean image for the training data. The GPU 206/218subtracts the mean image from each input image of the training set 434.The GPU 206/218 then performs feature standardization to make eachfeature in the dataset have a zero mean and a standard deviation of 1such that all image data features are normalized. Featurestandardization is used to ensure that measurement comparisons betweenfeatures that may have different units (such as audio signals and pixelvalues of the image data) are normalized. In feature standardization theimage features are centred on a zero mean with a standard deviationof 1. The mean image of the training set 434 is calculated. The meanimage is subtracted from each image in the training set 434. Theresulting value of each image is divided by its standard deviation. Theresulting value of each image feature is hence normalized and can befurther used for creating the training model.

The GPU 206/218 then defines the training model by selecting CNNarchitecture. In this example, the GPU 206/218 uses a CNN architecturemodel such as, Alexnet for defining the model. Alexnet is a CNNframework executed on GPUs implemented in CUDA. CUDA is a parallelcomputing platform and an application programming interface (API) modelcreated by Nvidia that can be used for general purpose processing alongwith graphics processing.

The model is then optimized using a solver algorithm. The solveralgorithm is a type of a stand-alone computer program or a softwarelibrary that is configured to optimize the training model by computingan accuracy of the model using the training dataset. The solveralgorithm computes the accuracy by iteratively using the validation set436. For example, the solver algorithm may use the validation set 436for every 1000 iterations in an optimization process of 40000 iterationsthat takes a snapshot of the trained model at every 5000^(th) iteration.

The GPU 206/218 then performs model training using the results of thesolver algorithm. During the training process, the GPU 210/240 monitorslosses and the model accuracy. In an example, Caffe takes a snapshot ofloss and model accuracy of the trained model at every 5000^(th)iteration. Then the GPU 210/240 plots a learning curve of the losses asa function of the number of iterations as depicted in the graph 438 ofFIG. 2B. Multiple iterations are performed until a steady-state accuracyrate is achieved. For example, as can be seen in the graph 438 thetrained model achieves an accuracy rate of about 90% that stopsimproving after about 3000 iterations.

Once the trained model is ready, the GPU 206/218 starts predictingaccuracy of unseen images from a testing dataset downloaded from aKaggle platform. The GPU 206/218 reads an image of an email from thetesting dataset, processes the image, and calculates a probability ofaccuracy, for example, 0 for non-business email, and 1 for a legitimatebusiness email. For example, if an accuracy rate of 98% is achieved,that image is considered to be of a legitimate business email.

In one embodiment, the model is stored in the cloud database of thecloud server 216 which is accessible by the mobile secretary applicationof the terminal device 200 via the network 220. In another embodiment,the model may be stored in the local memory 208 of the smartphone 200.The GPU 206 and/or 218 analyses an input image of an inbox email 420using the model. The image of the inbox email 420 is compared with thefeatures of a legitimate business email stored in the model. The GPU 206and/or 218 generates an output 245 that identifies the inbox email to bea legitimate business email 245 if the comparison yields a high accuracyrate. In this example, the GPU 206 and/or 218 identifies that the inputimage is of a legitimate business email that is related to a legalmatter. The GPU 206 and/or 218 identifies the business email 245 torelate to a court case the user has. The CPU 204 may compare historicaldata associated with the business email 245 stored in the local memoryof the mobile terminal 200 or the cloud server 216 to determine that theoutput, business email 245 relates to a court case that has the maintrial in 3 days in the calendar application. Hence, the CPU 204 mayrecommend the user to read the email right away. In an embodiment, thebusiness email 245 is placed on the idle screen scale of the mobileterminal 200. A draft response email may be drafted by the mobilesecretary cloud application into the draft folder of the emailapplication, for further editing and approval by the user. Subsequent tothe response, the email 245 is placed into folder “legal” in the emailapplication. The end result of this invention should be the computerizedreplica of the process where a human secretary looks at the email to seewhat it is like, and/or reads all or part of it, and then sorts it tothe relevant folder, and prepares obvious further action, for example adraft response to the arrived email.

By “machine learning” or “artificial intelligence” we mean that thecomputer system has been trained to make determinations based on atraining set of samples, and/or has been tested with a validation set tohave a known error rate as explained before.

Messages or emails can be classified either by analysing themsemantically, or visually as explained before. It is also in accordancewith the invention to use the semantic and the visual AI analysestogether to classify the messages or emails even more accurately. Thevisual AI analysis as explained in the above can be applied mutatismutandis to other secretarial tasks that would be done by the visualperception of the human secretary in accordance with the invention.Preferably the GPU carries out the visual AI tasks, and the CPU carriesout the semantic AI tasks in some embodiments of the invention.

Any features of embodiment 21 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 30, 31, 40, 41, 50, 60, 61, 62,63, 71, 81, and/or 82 in accordance with the invention.

Although the example discussed herein is provided with reference tousing convolutional neural networks that uses the deep learningframework Caffe, C++, MATLAB, and Python programming languages, theNVIDIA GPU, the Kaggle dataset, and the Alexnet CNN architecture model,it is to be understood that the mobile client 200 and the cloud server216, in another embodiment, may be implemented using any other deeplearning algorithm that uses any other framework, programming language,GPU, dataset, and/or architecture model. The example included herein isdescribed with reference to the publication “A PRACTICAL INTRODUCTION TODEEP LEARNING WITH CAFFE AND PYTHON”, which is included here as areference.

In another embodiment, to segregate the e-mails present in the e-mailapplication, the AI module 214 or the GPU 206 and/or 218 utilizes AI forprocessing of different details related to each e-mail. AI may be usedto process content of each e-mail, and to segregate each e-mail into arespective category. Processing the content of each e-mail may compriseprocessing of title, content, hyperlinks, dates, images, and attachmentspresent in each e-mail. Based on processing of such semantic content,the AI module 214 or the GPU 206 and/or 218 and/or CPU utilizing AI maysegregate each e-mail into a related category, so that emails areorganized into folders of different categories, and emails classified tobe in the same semantic category are placed in the same folder.

In one embodiment, the GPU may utilize “Caffe” as the AI for operatinganother software application. In some embodiments Processing of data bythe GPU running the AI may provide leverage for faster processing ofdata. For example, the messages may be simultaneously processed based ontheir title, words, hyperlinks, and dates present therein. The semanticAI analysis of the invention can be possibly implemented in someembodiments with a ready software package such as NLP, Leiki, Thron orsimilar semantic AI engine.

It is also in accordance with the invention to segregate emails based onimages taken of the emails only, based on the aforementioned contentanalysis of the emails only, or by using the two methods in a mix.

Any features of embodiment 10 may be readily combined or permuted withany of the other embodiments 11, 20, 21, 30, 31, 40, 41, 50, 60, 62, 63,71, 81, and/or 82 in accordance with the invention.

FIG. 1B demonstrates an embodiment 11 of a method of implementation of amobile secretary cloud application for operating a third softwareapplication, using another software application, as a flow diagram. Themethod could be implemented in a system identical or similar toembodiment 20 in FIG. 2 and embodiment 50 in FIG. 5 for example. Theend-user of the method could use a user interface identical or similarto that disclosed with embodiment 30, 31, 60, 61, 62, and 63 in FIG. 3A,FIG. 3B, FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D.

In phase 108, the data analyzing module 210 of the mobile secretarycloud application using the GPU 206 and/or 218 reads data from anothersoftware application. Another software application indicates a mobileapplication different from the mobile secretary cloud application.Reading the data signifies extracting all entries related to the userfrom another software application. For example, while a calendarapplication is accessed, all events scheduled in a particular month,dates and timings of such events, and priorities associated with suchevents are accessed. The events may comprise meetings, interviews,journeys, birthdays, anniversaries, etc. In one case, the GPU 206 and/or218 may read the data belonging to another software application fromRandom Access Memory (RAM) or cache memory. Further, metadata belongingto the data may also be fetched in one case.

In another case, an e-mail application may be accessed to retrieve alle-mails present in a mailbox of the user. The data analyzing module 210of the mobile secretary cloud application or the GPU 206 and/or 218 mayaccess the mail application. All mails present in the mailbox i.e.received, sent, transmission in progress, and recently deleted may beaccessed.

In phase 110, the data modifying module 212 of the mobile secretarycloud application or the GPU 206 and/or 218 operates a third softwareapplication based on data read from another software application. Thethird software application indicates a mobile application different fromthe mobile secretary cloud application and another software application.In one case, another software application may refer to the messagingapplication and the third software application may refer to the calendarapplication. The data modifying module 212 and/or the GPU 206, 218and/or processor may extract details related to a scheduled meeting,from at least one message stored in the messaging application. Based onextracted details, such as time and place of meeting/event, the datamodifying module 212 or the GPU 206, 218 and/or processor 204 mayschedule a reminder/event in the calendar application. Thereminder/event may be scheduled based on date and time of the meeting,or persons to be contacted or met in the meeting.

In another embodiment, the data modifying module 212 or the GPU 206, 218and/or processor 204 may extract details of a journey planned withfriends. Details of the journey may comprise a date and a flight numberfor booking seats for the journey. Based on extracted details, the datamodifying module 212 or the GPU 206, 218 and/or processor 204 may bookflight tickets using a related application. For example, the extracteddetails may comprise text mentioning “journey begins on Dec. 19, 2017 at9:30 AM from Delhi, India to New York, United States and returning backon Dec. 28, 2017.” Using such date, time, and place of origin anddestination, the data modifying module 212 or the GPU 206, 218 and/oroperator 204 may reserve flight tickets automatically.

In phase 112, the AI module 214 of the mobile secretary cloudapplication or the GPU 206, 218 and/or processor 204 utilizes ArtificialIntelligence (AI) to operate the third software application. The AImodule 214 or the GPU 206, 218 and/or processor 204 operates the thirdsoftware application for assisting a user and/or to perform a task forthe user. In one case, to identify and successively extract detailsrelated to the scheduled meeting, the AI module 214 or the GPU 206, 218and/or processor 204 utilizes AI for processing of different detailspresent in the at least one message. Processing the content of eachmessage may comprise processing of title, words, hyperlinks, dates andtimes, and passwords present in the at least one message. Based onprocessing of such content, the AI module 214 or the GPU 206, 218 and/orprocessor 204 may schedule the reminder/event in the calendarapplication.

In one case, the AI module 214 or the GPU 206, 218 and/or processor 204utilizes Artificial Intelligence (AI) to process the text “journeybegins on Dec. 19, 2017 at 9:30 AM from Delhi, India to New York, UnitedStates and returning back on Dec. 28, 2017.” Using AI, the AI module 214or the GPU 206, 218 and/or processor 204 may identify the date, time,and place of origin and destination details related to the journey. Thedata modifying module 212 or the GPU 206, 218 and/or processor searchesfor a flight matching the journey details. Successively, upon finding asuitable flight, the AI module 214 or the GPU 206, 218 and/or processorbooks a required number of flight tickets.

Choosing which ticket to buy, or present to user for purchase isdetermined similarly as explained before with classifying emails. Ahuman secretary would look at the emails, and see the typing, layout,language and everything to make a visual determination of whether theemail is a legitimate business email. Choosing a ticket is a slightlydifferent process. A human secretary would not look and determine thechoice by appearance. Instead, price, duration, departure time, arrivaltime and transit time in a stopover are the key factors in choosing whatticket to buy.

First a training set of e.g. 30,000 flight tickets is created. Avalidation set comprising ⅙^(th) of the flights is also created. Thenmobile secretary cloud application is taught what flights are acceptableand what are unacceptable. For example, with the validation set, onlythose fights with a low price, short transit time, short duration,coinciding departure and arrival times are selected. This is done forexample by calculating the normalized standard deviations for eachparameter and assigning them weights. The ticket that deviates the leastfrom an acceptable ticket is either bought or presented to the user forpurchase automatically.

Any features of embodiment 11 may be readily combined or permuted withany of the other embodiments 10, 20, 21, 30, 31, 40, 41, 50, 60, 61, 62,63, 71, 81, and/or 82 in accordance with the invention.

FIG. 3A demonstrates an embodiment 30 of software program product userinterfaces 300 and 306 in accordance with the invention as a screen shotdiagram. The user interfaces show the mobile secretary cloud application302, 308 operating another software application. The user interfacescould be displayed for example on a display screen of a mobile clientdevice i.e. a smartphone.

The software program product is stored on a non-transient memory mediumi.e. the memory 208 of the system i.e. mobile client device 200 and thecloud server 216. The software program product may also be distributedbetween the memory 208 of the mobile client device 200 and the cloudserver 216, so that some parts of the software program product reside inthe memory 208 and some parts of the software program product reside onthe cloud server 216.

In an embodiment, a software program product user interface 300 is shownto comprise a mobile secretary cloud application 302 operating anothersoftware application i.e. a messaging application 304. The mobilesecretary cloud application 302 reads data from the messagingapplication 304 and operates the messaging application 304 based on readdata. The read data may include contents of messages, details of sendersof the messages, date and time of receipt of the messages etc. All suchdetails may be processed using AI and accordingly, the messagingapplication may be operated. In one case, a message may be received froma colleague and the message may ask a user to call his colleague. Duringsuch instance, if it is determined that the user is driving a vehicle,the messaging application may automatically answer to the message thatthe user is driving and that the user shall get in touch after a fewminutes. The user driving the vehicle could be determined based onaccelerometer data of the mobile client device of the user or based on adriving mode set by the user on the mobile client device or fromlocation changes of the mobile client device based on GPS or basestation measurements. Similarly, it is also possible to use images ofemails to determine their type and characteristics as explained before.

In an embodiment, a software program product user interface 306 is shownto comprise a mobile secretary cloud application 308 operating anothersoftware application i.e. a calling application 310. The mobilesecretary cloud application 308 may interact with the callingapplication 310 on user's behalf. In one case, the calling application310 may be connected to an Interactive Voice Response System (IVRS) forbooking flight tickets for the user. The mobile secretary cloudapplication 308 may utilize AI for analyzing all instructions receivedvia the IVRS. For example, the user may be required to set a preferredcommunication language over the IVRS by pressing of a soft key. Themobile secretary cloud application 308 may determine English aspreferred communication language of the user, based on previouscommunication of the user over calls, language of content present inmessage, mails, etc. In one case, the mobile secretary cloud applicationmay analyze user's interaction with the IVRS over several calls, tolearn user's preferences and other user data provided by the user, overthe IVRS. The other user data may comprise personal details, paymentdetails, and details for user identity verification. Successively, themobile secretary cloud application 308 may respond to select English asthe preferred language of the user by selecting the soft key assigned toEnglish. Thereafter, the mobile secretary cloud application 308 mayprovide other inputs to the IVRS, such as providing personal details ofthe user, answering to authentication questions for the user, andselecting a particular department to be contacted via the IVRS either byemulating soft key presses, or reading out answers with synthesizedvoice down the phone line. Thus, taking the burden of interacting withthe IVRS, or a human service agent, when the answer to the questionasked by the IVRS or the human service agent is known.

In one preferred embodiment, the mobile secretary application gets ahuman service agent on the line. Remember in the old times, whensecretaries would get people “on the line”, i.e. waiting to thetelephone for the CEO?—The mobile secretary cloud application can dothat for the normal consumer. The mobile secretary cloud application 308simply navigates through the choices of the IVRS with selections thatdeviate the least from past selections of the user or users that havebeen recorded. Further, the mobile secretary cloud application mayidentify questions asked by the human service agent, such as “What isyour Date of Birth?” or “What is your Customer number?”, andsubsequently voice recognize these questions, and then dictate the dateof birth of the user, or the customer number of the user down the phoneline with synthesized voice.

Similarly to before, a training set of maybe 30,000 IVRS discussions canbe created. The mobile secretary cloud application and cloud network isthen taught with 5000 IVRS validation set discussions, what is asatisfactory conduct of the IVRS discussion. Typically, this is the IVRSdiscussion that deviates the least from the facts available to themobile secretary cloud application, and the mobile secretary cloudapplication will attempt to conduct that IVRS discussion with a computeror a human service agent on behalf of the user via the phone line to asfar as possible. When the mobile secretary cloud application can nolonger determine a satisfactory response, it may consult the user ordirect that part of the interaction to the human user so that he mayconduct that part of the interaction himself. The mobile secretary cloudapplication may also contain goal based standard call & discussionprotocols, for example “get window seats for flight”. When selected, themobile secretary cloud application calls the airline with the bookingreference obtained from the emails, and conducts the discussion withsoft keys or synthesized voice to obtain a window seat for the user. Ifthat does not work, the mobile secretary cloud application 302, 308, 314may automatically try to obtain the window seat via airline webpages byfilling on-line forms of the airline or an online travel agent, andoptionally purchasing tickets thereby with the user's credit carddetails stored in the cloud network or the mobile terminal. Of course,the order in which the call and the website are used could be reversed:the mobile secretary cloud application could try first to obtain thewindow seat via the webpage, but if that does not work it could call theairline as explained before to obtain a window seat. For activities likethis separate training sets and validations sets could be configured,for example by recording 30,000 phone conversations where a window seatis obtained, or by recording 30,000 web sessions where a window seat isobtained.

However, the sizes of both the training set and validation can be grownto much higher sample numbers. Especially if the quality of the samplesis poor and/or the data samples are small are higher numbers preferable.It is known from A. Halevy, P. Norvig and F. Pereira The UnreasonableEffectiveness of Data, IEEE Intelligent Systems 8-12, 2009 that atrillion messy data samples may be more useful in machine learning thana clean data set of million items. This document is cited here asreference. It is in accordance with the invention to use sample setsizes in the millions, billions and trillions of samples.

Any features of embodiment 30 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 31, 40, 41, 50, 60, 61, 62, 63,71, 81, and/or 82 in accordance with the invention.

FIG. 3B demonstrates an embodiment 31 of a software program product userinterface 312 in accordance with the invention as a screen shot diagram.The user interface 312 shows the mobile secretary cloud application 314utilizing another software application for operating a third softwareapplication. The user interface could be displayed for example on adisplay screen of a mobile client device i.e. a smartphone.

In an embodiment, a software program product user interface 312 is shownto comprise a mobile secretary cloud application 314 utilizing anothersoftware application for operating the third software application, whereanother software application and the third software application isselected from a messaging application 316, a calling application 318,and a map application 320. In one embodiment, the mobile secretary cloudapplication 314 accesses data present in the messaging application 316for operating the calling application 318. For example, the messagingapplication may comprise a message including calling details of an AirConditioning (A.C.) repairing agency. Upon retrieving such details fromthe message, the mobile secretary cloud application 314 mayautomatically call the A.C. repairing agency and may answer to an IVRSsystem over the call, for arranging repair of the A.C. In another case,the messaging application may comprise another message including detailsof a venue for attending a meeting by the user. Based either on a user'srequest or automatically, on the date of the meeting, the mobilesecretary cloud application 314 may operate the map application 320 forsetting a navigation path, to guide the user for reaching the venue ofmeeting. In some embodiments this navigation path is sent to aself-driving car configured to transport the user.

Any features of embodiment 31 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 30, 40, 41, 50, 60, 61, 62, 63,71, 81, and/or 82 in accordance with the invention.

FIG. 4A illustrates an embodiment 40 of a flow chart showing a method ofimplementation of a mobile secretary cloud application for operatingseveral other software applications. The different phases/stepsmentioned in FIG. 4 a are not necessarily performed in mentionedsequence, but could be performed in different sequences orindependently.

FIG. 5 illustrates an embodiment 50 of a block diagram of a system 500executing the mobile secretary cloud application. The system comprisesinterface(s) 502, processor 504, Graphical Processing Unit (GPU) 506,and memory 508. The memory 508 comprises a mail and message processingmodule 510, a call management module 512, and a navigation module 514.In an embodiment, the system 500 is integrated with a cloud server 516,via a communication network 520. The cloud server 516, the GPU 518, andthe communication network 520 are similar to the cloud server 216, GPU218, and the communication network 220 of FIG. 2 respectively. Differentphases of FIG. 4A are now explained in conjunction with modules of FIG.5 .

Any features of embodiment 50 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 60, 61, 62,63, 71, 81 and/or 82 in accordance with the invention.

In phase 402, the mobile secretary cloud application replicatessecretary functions i.e. the mobile secretary cloud application performstasks for the user and/or provides assistance to the user in managingtasks by operating other software application(s) installed on the mobileclient device like a human secretary would.

In phase 404, another software application is an e-mail or messagingapplication 604. The e-mail and message processing module 510 of themobile secretary cloud application 602 or the GPU 506, 518 and/orprocessor 504 analyses messages and e-mails, and automatically sorts themessages and e-mails into different folders. Analysis of the messagesand e-mails may be performed by utilizing AI, that processes contents ofthe messages and e-mails and other information related to the messagesand e-mails, for sorting the messages and e-mails into different folders606 a-606 c. The artificial intelligence can be realized by teaching themobile secretary cloud application to distinguish an email from theimage of the email, with training sets and validations sets of images ofemails for example as explained before. Alternatively or additionally,the AI can classify an email based on metadata and semantic content, inwhich case a training set of emails and validations set of emails wouldbe used so, that the artificial intelligence secretary application istaught to classify emails based on their semantic content, and metadata,such as time, recipient, sender and the like. Further these two AImethods can be used together in a mix.

In phase 406, the e-mail and message processing module 510 or the GPU506, 518 and/or processor 504 analyses messages and selects anindication sound (amongst 616 a-616 c) automatically based onclassification, contents of the messages and sender and/or recipients ofthe messages. The e-mail and message processing module 510 or the GPU506, 518 and/or processor 504 may utilize AI to process contents of themessages and may select an indication sound based on the contents. Forexample, the mobile client device of the user may receive a pleasantmessage, stating “You did an incredible job.” Similarly, the user mayreceive other pleasant messages, such as “Thank you for your kindsupport,” “I am glad to hear about your promotion,” and “love you myson.” The e-mail and message processing module 510 or the GPU 506, 518and/or processor 504, utilizing AI, may learn upon analyzingsemantically several such messages, and may determine that such messagesconvey pleasant gestures.

The e-mail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may be taught to learn or know the semantic meanings ofwords present in the e-mails and messages. For example, in oneembodiment, the words “incredible,” “thank you,” “kind,” “glad,”“promotion,” and “love” infer pleasant gestures from a sender, and thee-mail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may associate a positive meaning to such words. Based onsuch learning, the e-mail and message processing module 510 or the GPU506, 518 and/or processor 504 may play uplifting and positive audio asthe indication sound.

In a contrasting case, the mobile client device of the user may receivea harsh message, stating “poor job.” Similarly, the user may receiveother harsh messages, such as “you've passed the deadline,” “I won'ttolerate such careless attitude,” and “hate you.” The e-mail and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 utilizingAI, may learn upon several such messages, and may determine that suchmessages convey harsh and negative gestures. The e-mail and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 may betaught to attribute a negative meaning to these words present in thee-mails and messages. For example, in the present embodiment, the words“poor,” “passed deadline,” “won't tolerate,” and “hate” infer harsh andnegative gestures from a sender, and the e-mail and message processingmodule 510 using the GPU 506, 518 and/or processor may be taught to knowor learn such semantic meaning. Based on such learning or knowledge, thee-mail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may play a loud beep or warning signal as the indicationsound.

Further, the e-mail and message processing module 510 or the GPU 506,518 and/or processor 504 may also derive meanings based on analysis ofsmileys/emoticons and special characters present in the e-mails andmessages. The special characters may include “!!,” “?” and other knownspecial characters. For example, the user may receive a message stating“you again!!” or “what have you done?” The e-mail and message processingmodule 510 or the GPU 506, 518 and/or processor 504 may learn and/ordistinguish the delivery of a harsh gesture through such message. Basedon such learning, the e-mail and message processing module 510 or theGPU 506, 518 and/or processor 504 may play a warning sound or a loudbeep as the indication sound.

In phase 408, another software application is a telephony application626. Using the telephony application 626, the call management module 512of the mobile secretary cloud application 622 or the GPU 506 and/or 518may make phone calls on behalf of the user, may automatically answer tomessages or compose draft replies, and may answer to incoming calls onbehalf of the user, and may prompt queries and answer to queries onbehalf of the user, using a synthesized voice on a phone line. Forexample, during an IVRS phone call the user's phone number and customernumber may be asked. The call management module 512 or the GPU 506, 518and/or processor 504 may recognize the asked questions by voicerecognition software and provide answers to the recognized questionsbased on previous calls or data available in the emails, messages, orthe filesystem, and may respond to the IVRS or customer attendant in thecall center by reading the information using synthesized voice.

In another embodiment, the user may need to book flight tickets fromMumbai, India to New York, United States and details of such journey maybe stored in an e-mail or message. Based on approval of the user orautomatically, the mobile secretary cloud application 622 accesses suchdetails for booking the flight tickets for the user. The call managementmodule 512 or the GPU 506 and/or 518 may call a customer care number ofan airline, use a voice synthesizer to read out and voice recognitionsoftware to recognize details for communicating over the call, and thusbook the flight tickets over the phone.

In phase 410, another software application is a map application 634. Thenavigation module 514 of the mobile secretary cloud application 632using the GPU 506, 518 and/or processor 504 using the map application634 provides a route calculator that provides a combined route ofdriving and walking based on e-mails, calendar entries, and locationsearch. The navigation module 514 or the GPU 506, 518 and/or processor504 upon accessing data present in the e-mails and the calendar entriesmay determine a date and time of reaching a location by the user. Thenavigation module 514 or the GPU 506, 518 and/or processor 504 maydetermine multiple routes for reaching the location. Further, thenavigation module 514 or the GPU 506, 518 and/or processor 504 mayaccess recent location searches performed by the user, and may utilizesuch information for filtering the multiple routes. In one case, thenavigation module 514 or the GPU 506, 518 and/or processor 504 mayprovide the user with a route which is most familiar to the user, i.e.the user has been through at least once.

The method explained with Graph 438 in FIGS. 2A and 2B and elsewhereearlier in the description can be used with all third applicationse-mail 510, calls 512, navigation 514. The mobile secretary cloudapplication and cloud server 516 is first provided with a large trainingset, spanning the diversity of emails, phone conversations, ornavigation routes. The mobile secretary cloud application is then taughtwith a validation set of emails, phone conversations or navigationroutes what is an acceptable choice for a user. Once the error rate isdiminished below a threshold with a large number of iterations as shownin graph 438 of FIG. 2B, the model is deployed to users. Naturallymachine learning files can be continuously maintained, updated andimproved on the cloud server network 516 from which the machine learningfiles and their updates are made available to the terminal devices 200,500.

Any features of embodiment 40 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 41, 50, 60, 61, 62,63, 71, 81, and/or 82 in accordance with the invention.

FIG. 4 b illustrates a flow chart showing a method of implementation ofa mobile secretary cloud application for operating several thirdsoftware applications, using the intelligent secretary softwareapplication, according to the invention. The different phases/stepsmentioned in FIG. 4 b are not necessarily performed in mentionedsequence, but could be performed in different sequences orindependently. Different phases of FIG. 4 b are now explained inconjunction with modules of FIG. 5 .

In phase 412, the mobile secretary cloud application replicatessecretary functions i.e. the mobile secretary cloud application performstasks for the user and/or provides assistance to the user in managingtasks by operating software application(s) installed on the mobileclient device like a professional secretary would.

In phase 414, the third software application operated by the mobilesecretary cloud application 602 is an e-mail or messaging application604. The e-mail and message processing module 510 or the GPU 506, 518and/or processor 504 of the mobile secretary cloud application 602analyses messages and automatically sorts the messages to differentfolders 606 a-606 c shown in FIG. 6A. Along with information present inthe messages and e-mails, the e-mail and message processing module 510or the GPU 506, 518 and/or processor 504 may utilize data of anotherapplication, for example a calendar application, for sorting themessages and e-mails to different folders 606 a-606 c. In one embodimentthe folders 606 a-606 c have different urgency levels, like one day, oneweek, one month and so on. The mobile secretary cloud application movesthe emails to the different folders 606 a-606 c and in between thefolders 606 a-606 c as the emails become more or less urgent, as newentries are marked into the calendar, and so forth.

In phase 416, the e-mail and message processing module 510 or the GPU506, 518 and/or processor 504 analyses messages and selects anindication sound (amongst 616 a-616 c) automatically based on contentsof the messages and sender and/or recipients of the messages. In onecase, the e-mail and message processing module 510 or the GPU 506, 518and/or processor 504 may utilize AI to process contents of the messagesand may select an indication sound based on the contents, or the folderto which the message is classified.

In phase 418, the third software application is a telephony application626. Using the telephony application 626, the call management module 512or the GPU 506, 518 and/or processor 504 of the mobile secretary cloudapplication 622 makes phone calls on behalf of the user, mayautomatically answer to messages or incoming phone calls, and may answerto queries on behalf of the user, using a synthesized voice on the phoneline. For example, during an IVRS phone call the user's phone number andcustomer number may be asked. The call management module 512 mayrecognize the question by voice recognition software and find requestedinformation in a digital wallet of the user or other application storingsuch detail, and may respond on the IVRS by reading the informationusing the synthesized voice. In another embodiment, the user may need tobook flight tickets from Mumbai to New York and details of such journeymay be stored in an e-mail or message. Based on approval of the user,the call management module 512 or the GPU 506, 518 and/or processor 504access such details for booking the flight tickets for the user. Thecall management module 512 or the GPU 506, 518 and/or processor 504 forexample calls a customer care number of an airline, uses a voicesynthesizer to read such details for communicating over the call, andthus books the flight tickets. Alternatively, the mobile secretary cloudapplication may be configured to operate an e-commerce application tobook the flights. The acceptable flight or acceptable flights forpurchase can be determined as explained before, by providing a trainingset and validation set of flights and teaching the mobile secretarycloud application to select an acceptable flight that, for exampledeviates from the model acceptable flight the least in a number ofparameters such as duration, price, departure and arrival time and/orstopover duration.

In phase 419, the third software application is a map application 634.The navigation module 514 of the mobile secretary cloud application orthe GPU 506, 518 and/or processor 504 using the map application providesa route calculator that provides a combined route of driving and walkingbased on e-mails, calendar entries, and location search performed by theuser for conducting the activities of the day at various locations andtimes. The navigation module 514 or the GPU 506, 518 and/or processor504 upon accessing data present in the e-mails 636 and the calendarentries 638 may determine a date and time of reaching a location by theuser. The mobile secretary cloud application may determine multipleroutes for reaching the location. Further, the navigation module 514 orthe GPU 506, 518 and/or processor 504 may access recent locationsearches 640 performed by the user, and may utilize such information forfiltering the multiple routes. In one case, the navigation module 514 orthe GPU 506, 518 and/or processor may provide the user with a routewhich is most familiar to the user, or which route when followed allowsthe user to complete the maximum number of errands or tasks on his to dolist.

The aforementioned secretarial process of optimizing routes, agendasand/or timetables naturally results in the knowledge of events that theuser has actually visited. The location and time of the mobile stationcan be followed and calendar entries and Internet search can beharnessed along with call application data to construct a history ofevents that the user has attended. Many of these events are likely to berecurring: conferences happen every year, the barber has to be visitedonce per month, the dentist once per year, the accountant needs to becontacted every 3 moths to prepare a quarterly report and so forth. Themobile secretary cloud application can automate the rebooking of theuser into these events. In the first stage, the mobile secretary cloudapplication 602, 612, 622, 632 may populate the calendar with entrieswhen the event takes place or should take place. The mobile secretarycloud application 602, 612, 622, 632 may search the Internet todetermine the 2019 dates of a conference that the user attended in 2018,and mark them up in the calendar. The Mobile secretary cloud application602, 612, 622, 632 may also register the user to this conferenceautomatically or upon approval of the user. The mobile secretary cloudapplication may book barber and dentist appointments at a convenienttime and place automatically or upon approval of the user as explainedbefore via phone or the webpage of the barber or dentist. All of theseactivities can be done without or almost without any user participationby the mobile secretary cloud application merely repeating recordedtelephone conversations or webpage sessions and adapting them with AI tothe current or upcoming situation. If plans change the mobile secretarycloud application 602, 612, 622, 632 may automatically cancel activitiesor events on behalf of the user, by similarly executing web sessions orphone conversations of cancellation on behalf of the user, adapted withAI to the current situation.

Any features of embodiment 41 may be readily combined or permuted withany of the other embodiments 10, 20, 21, 30, 31, 40, 50, 60, 61, 62, 63,71, 81, and/or 82 in accordance with the invention.

FIG. 6A demonstrates an embodiment 60 of a software program product userinterface 600 in accordance with the invention as a screen shot diagram.The user interface 600 shows the mobile secretary cloud application 602operating an e-mail/messaging application 604. The user interface 600could be displayed for example on a display screen of the mobile clientdevice i.e. a smartphone 500.

In an embodiment, the software program product user interface 600 isshown to comprise the mobile secretary cloud application 602 operatingthe e-mail/messaging application 604. The mobile secretary cloudapplication 602 reads data from the e-mail/messaging application 604 andoperates the e-mail/messaging application 604 based on read data. Theread data may include contents of messages, details of senders of themessages, date and time of receipt of the messages etc. All such detailsmay be processed using AI and accordingly, the messaging application maybe operated. In one case, based on processing of the read data, themobile secretary cloud application 602 may store official work-relatedmails and messages into folder 606 a, e-mails and messages from friendsand families into folder 606 b, and other e-mails and messages, such asfrom banks, telecom operators, and other service providers into folder606 c. For example, the messages and e-mails may be sorted intodifferent folders based on the senders, month of receipt, and details ofthe messages/e-mails. The mobile secretary cloud application may alsotake images or photographs of the emails to graphically process theemails as explained before in association with FIGS. 2A and 2B.Similarly, the photographic/visual classification of e-mails or messagesmay be done together with the semantic classification so that these twomethods are used in a mix. It is also in accordance with the inventionfor the mobile cloud secretary application to search the Internet togain further information about sender, recipient or the like to assistin the message and/or email classification.

Any features of embodiment 60 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 61, 62,63, 71, 81, and/or 82 in accordance with the invention.

FIG. 6B demonstrates an embodiment 61 of a software program product userinterface 610 in accordance with the invention as a screen shot diagram.The user interface 610 shows the mobile secretary cloud application 612operating an e-mail/messaging application 614. The user interface 610could be displayed for example on the display screen of the mobileclient device i.e. a smartphone.

In an embodiment, the software program product user interface 610 isshown to comprise the mobile secretary cloud application 612 operatingthe e-mail/messaging application 614. The mobile secretary cloudapplication 612 analyses messages and selects an indication soundautomatically based on contents of the messages and sender and/orrecipients of the messages. In one case, the e-mail and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 utilizesAI to process contents of the messages and selects an indication soundbased on the contents. For example, the mobile client device of the usermay receive a pleasant message, stating “You did an incredible job.”Similarly, the user may receive other pleasant messages, such as “Thankyou for your kind support,” “I am glad to hear about your promotion,”and “love you my son.” The e-mail and message processing module 510 orthe GPU 506, 518 and/or processor 504, utilizing AI, may be taught thatthe semantic meaning of such messages convey pleasant gestures. Thee-mail and message processing module 510 or the GPU 506, 518 and/orprocessor 504 may also continuously learn and update machine learningfiles, or deep learning files or Artificial Intelligence files on theterminal or in the cloud, based on processing of words present in thee-mails and messages. Typically, in some embodiments these machinelearning files, or deep learning files or Artificial Intelligence filescontain training set and/or validation set data. For example, in presentembodiment, the words “incredible,” “thank you,” “kind,” “glad,”“promotion,” and “love” infer pleasant gestures from a sender, and thee-mail and message processing module 510 or the GPU 506, 518 and/orprocessor may learn such meaning, and/or attribute such semantic meaningto these words. Based on such learned or attributed meaning, the e-mailand message processing module 510 or the GPU 506, 518 and/or processor504 may play a positive and uplifting audio 616 a with a loudspeaker asthe indication sound.

It is in accordance with the invention to have the Artificialintelligence files used to train and validate the ArtificialIntelligence process on the terminal, in the cloud and/or distributedbetween both of them. Artificial intelligence files, deep learningfiles, machine learning files and self-learning files are usedinterchangeably, and refer to the files used to establish the trainingand validation process producing the artificial intelligence asexplained before for example with FIG. 2B.

In a contrasting case, the mobile client device of the user may receivea harsh message, stating “poor job.” Similarly, the user may receiveother harsh messages, such as “you've passed the deadline,” “I won'ttolerate such careless attitude,” and “hate you.” The e-mail and messageprocessing module 510 or the GPU 506 and/or 518, utilizing AI, may learnupon several such messages, and may determine that such messages conveydissatisfaction from the sender. The e-mail and message processingmodule 510 or the GPU 506, 518 and/or processor 504 may alsocontinuously learn based on processing of words present in the e-mailsand messages, and update the machine learning files on the terminaldevice and/or the cloud server. For example, in present case, the words“poor,” “passed deadline,” “won't tolerate,” and “hate” inferdissatisfaction from a sender, and the e-mail and message processingmodule 510 or the GPU 506, 518 and/or processor may know, or learn suchmeaning. Based on such learning or knowing the semantic meaning, thee-mail and message processing module 510 using the GPU 506, 518 and/orprocessor 504 may play a loud beep 616 b or warning sound as theindication sound with a loudspeaker.

Further, the e-mail and message processing module 510 or the GPU 506,518 and/or processor 504 may also derive meanings based on analysis ofsmileys/emoticons and special characters present in the e-mails andmessages. The special characters may include

“!!,” “?” and other known special characters or emoticons. For example,the user may receive a message stating “you again!!” or “what have youdone?” The e-mail and message processing module 510 or the GPU 506, 518and/or processor may learn delivery of harsh gesture through suchmessage, or a positive gesture if the emoticon has a positive semanticmeaning. Based on such knowing or learning, the e-mail and messageprocessing module 510 or the GPU 506, 518 and/or processor 504 may playa positive uplifting signal for positive emoticons and a negative audiosignal such as a loud beep 616 b as the indication sound for negativeemoticons. Similarly, other indication sounds 616 c may be used fordifferent special characters or emoticons indicating other expressionsand/or gestures included in the e-mails and/or messages received by theuser.

Any features of embodiment 61 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 62,63, 71, 81, and/or 82 in accordance with the invention.

FIG. 6C demonstrates an embodiment 62 of a software program product userinterface 620 in accordance with the invention as a screen shot diagram.The user interface 620 shows the mobile secretary cloud application 622operating the calling application 626. The user interface 620 could bedisplayed for example on the display screen of the mobile client devicei.e. a smartphone.

In an embodiment, the software program product user interface 620 isshown to comprise the mobile secretary cloud application 622 operating acalling application 626. The mobile secretary cloud application 622 mayinteract with the calling application 626 on user's behalf. In one case,the calling application 626 may be connected to an Interactive VoiceResponse System (IVRS) for bookings flight tickets for the user. Themobile secretary cloud application 622 may utilize AI for analyzing allinstructions or voice prompts received via the phone line from the IVRS.For example, the user may be required to set a preferred communicationlanguage over the IVRS by pressing of a soft key or number key. Themobile secretary cloud application 622 may determine English aspreferred communication language of the user, based on user data anduser preferences 624 of the user. Successively, the mobile secretarycloud application 622 may respond to select English as preferredlanguage of the user. Thereafter, the mobile secretary cloud application622 may provide other inputs to the IVRS, such as providing personaldetails of the user, answering to authentication questions for the user,and selecting a particular department to be contacted via the IVRS. Themobile secretary cloud application 622 may retrieve all requiredinformation from the user data and user preferences 624 of the user. Forexample, the user data may comprise travelling details, personal detailsand payment details of the user. Thus, the mobile secretary cloudapplication 622 may provide the travelling details over the IVRS and mayfinally provide the payment details, over the IVRS, for booking a flightticket for the user.

Any features of embodiment 62 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,63, 71, 81, and/or 82 in accordance with the invention.

FIG. 6D demonstrates an embodiment 63 of a software program product userinterface 630 in accordance with the invention as a screen shot diagram.The user interface 630 shows the mobile secretary cloud application 632operating the map application 634. The user interface 630 could bedisplayed for example on the display screen of the mobile client devicei.e. a smartphone.

In an embodiment, the software program product user interface 630 isshown to comprise the mobile secretary cloud application 632 operatingthe map application 634. The mobile secretary cloud application 632using the map application 634 provides a route calculator that providesa combined route of driving and walking based on e-mails, calendarentries, and location search. The e-mails may be accessed from thee-mail/messaging application 636, calendar entries may be accessed fromthe calendar application 638, and location search performed by the usermay be accessed from a location search application 640 or Internetbrowser. The mobile secretary cloud application 632 upon accessing datapresent in the e-mails and the calendar entries may determine a date andtime of reaching a location by the user. The mobile secretary cloudapplication 632 may determine multiple routes for reaching the location.Further, mobile secretary cloud application 632 may access recentlocation searches performed by the user, and may utilize suchinformation for filtering the multiple routes. In one case, the mobilesecretary cloud application 632 may provide the user with a route whichis most familiar to the user, i.e. the user has been through at leastonce.

In one case, the mobile secretary cloud application 632 may identifymultiple calendar entries present in the calendar application 638, for aparticular day. The user may need to visit multiple locations forattending different meetings scheduled on the day. In such case, themobile secretary cloud application 632, using the map application 634may provide a route map to the user, such that a least distance or timewill need to be travelled to visit all the locations.

Any features of embodiment 63 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,62, 71, 81, and/or 82 in accordance with the invention.

In the preceding description it has been described how the intelligentsecretary mobile application arrives at good or optimum outcomes insecretarial tasks using machine learning and artificial intelligence. Asexplained before, this is typically achieved by taking a training set ofsubstantial size, typically 30,000 samples or so, and teaching thecomputer with a validation set of about 5000 samples what is anacceptable choice and what is not an acceptable choice. This principleis applied to different secretarial tasks regardless of the data formatof the output. For example, if the secretarial task is looking at emailsand judging which emails are genuine and important, the sample sets usedwill be semantic data. i.e. words and content of the email, its metadata(sender, recipient, time, title) and also optionally the image of theemail. The type setting, colors, layout, all visual factors contributeto the decision of how to characterize the email.

If the secretarial task is interaction on the phone, the training setsand validation sets will likely be audio soundtrack and dial toneentries. If the secretarial task is route planning, the training andvalidation sets will be the routes. If the secretarial task is bookingflight tickets, the training and validations sets will be flighttickets.

Typically, the mobile secretary cloud application 602, 612, 622, 632,will be conducting Internet searches from time to time automatically tofind a result(s) that would be satisfactory to the AI model in theterminal device and cloud network. This means that if a good enoughroute, flights, or the like is not found, the secretary application cancontinue to conduct Internet searches to find a satisfactory result forthe user on his behalf.

In the aforementioned description, the mobile secretary cloudapplication 602, 612, 622, 632 has been described, and how the mobilesecretary cloud application arrives at good or optimum outcomes insecretarial tasks using machine learning and artificial intelligence. Itis also in accordance with the invention to combine optimized results ofindividual secretarial outcomes to arrive at an optimized agenda for anentire day for the user. This is achieved as follows. The terminalcomputer 200, 500 and cloud network 216, 516 is presented with atraining set of 30,000 or so daily agendas, and a validation set of afew thousand agendas to teach the mobile secretary cloud application602, 612, 622, 632 and the cloud server network 216, 516 to distinguishbetween a good daily agenda for the user, and an unacceptable agenda forthe user. When the mobile secretary cloud application 602, 612, 622, 632determines the individual outcomes to the secretarial tasks, it willthen fit these tasks to a daily agenda. This daily agenda will then becompared with the AI model, and the agenda that is accepted by thetrained AI model is presented to the user with the secretarial outcomesin accordance with the agenda. In some embodiments an agenda for a week,month and/or year, or any unit of time is similarly composed, using anAI model as explained before mutatis mutandis.

For each secretarial task, the mobile secretary cloud application 602,612, 622, 632 can be trained with deep-learning datafiles. These deeplearning data files may reside on a cloud server to which the clientdevice, i.e. typically a mobile phone is connected to via Wi-Fi orcellular connection. The mobile secretary cloud application 602, 612,622, 632 can be trained to conduct new secretarial tasks on behalf ofthe user by providing new sets of deep learning data files. Similarly,secretarial activities that the mobile secretary cloud application 602,612, 622, 632 already performs can be improved by updating deep learningdata files. The training and validation datasets for any activity can beupdated from time to time so that the mobile secretary cloud application602, 612, 622, 632 recognizes the need for an appropriate secretarialtask with the best accuracy, and that the mobile secretary cloudapplication 602, 612, 622, 632 fulfills the task with the best accuracy.

The mobile secretary cloud application 602, 612, 622, 632 may recordtasks that it performs either in the form of a computing session, arecording of a synthesized phone conversation, and/or text, and/orimages of messages or sessions that the mobile secretary cloudapplication 602, 612, 622, 632 performs. The recordings of past tasksmay be used to enrich the deep learning data files. The deep learningdata files may reside on either the client device such as the mobilephone or in the cloud in accordance with the invention. The deeplearning data files may be shared by one or more cloud servers betweendifferent users in some embodiments. In some embodiments the origin ofthe deep learning data files used to train the mobile secretary cloudapplication 602, 612, 622, 632 is concealed from the user, in order toprotect the privacy of those users from whom the deep learning datafiles originate. It is also in accordance with the invention that everyuser generates all or some of their own deep learning data files.

FIG. 7 demonstrates a machine-learning embodiment 71 of the invention asa flow diagram. In phase 712 the mobile cloud secretary applicationrecords user inputs into a cognitive software module. The purpose of therecording is to establish a baseline profile for the cognitive processof the user. The cognitive process here means how the user comprehendsthe incoming messages and changes in applications in the mobile phone,and how he responds and interacts with to the messages or applicationchanges. The cognitive module records for example keypresses,application sessions, user input, and user responses to certainsituations in the operating system or applications. Suppose for examplea calendar entry email, or an email suggesting an appointment to pick upthe kids comes into the inbox. The cognitive module records that theuser always enters it into his calendar very briefly as “pick up kids”.The user does not elaborately describe or explain the circumstances ofthe pickup.

In phase 714 the mobile secretary cloud application repeats newinstructed activities as recorded in the cognitive module, therebyreplicating work of the user for the user. Thus, when the appointmentemail comes into the inbox, the mobile secretary cloud applicationsimply makes the calendar entry “pick up kids” into the calendarapplication of the user, in line with the detected cognitive process ofthe user. It is easy for the user to comprehend this calendar entry, asit looks like he made it himself. This is a big advantage to justaccepting a calendar entry that someone else wrote, which might beincomprehensible, and probably also longer in text.

In phase 716 the mobile secretary cloud application derives aself-improved activity from crowd sourced data. As the crowd sourceddata shows that many people choose to have an alert set up 2 hoursbefore when something is entered into calendar about kids, the mobilesecretary cloud application introduces this improvement automatically.

The self-improved activity is typically determined by machine learningfrom crowd sourced data. This can be done by choosing an inputpresentation, i.e. the variable from which the best response dependsfrom in the crowd sourced data. Then a regression model is typicallyfitted with the crowd sourced data. For example here, the duration ofthe time interval before the actual event could be fitted against thesatisfaction rate of the alert (i.e. the rate at which the users approveand use the alert, without changing it to another time). Preferably, thealert interval providing the highest satisfaction rate from users isselected as the alert interval for the self-improved activity.

Different methods may be used in the mobile cloud secretary applicationtrying to improve and rationalize its operation with self-improvedactivities with creative suggestions on the one hand, and common senseinducing machine learning features on the other hand. Typicallycrowd-sourcing is used to introduce common senseness into theself-improvements. A wide user base is unlikely to use softwarecompletely irrationally.

Creativeness is introduced by supervised learning and/or reinforcementlearning. For example, in the reinforcement learning example, the alertinterval could be chosen based on an approximation of the maximumprobability at which the user will accept the self-improvement withoutmodifications, based on crowdsourced data.

In the supervised learning example, the alert could be defined so theuser has sufficient time, with for example a buffer of 30 minutes fromthe location of the user, to arrive at the pickup location of kids topick them up. The mobile secretary cloud application will know that theoutcome of the alert interval will need to be such that the alert willprovide sufficient time for the user to arrive between 0-30 minutesbefore to the pick-up location of the kids. As the cloud secretaryapplication follows the locations and routes of the user to the pick-uplocation every day, and for example the general traffic information, thecloud secretary application learns to position the alert in space andtime so that the user embarks on his journey to pick up the kids at theright time and is never late. This embodiment saves a lot of planningeffort, time and/or frustration for the human user.

In phase 718 the mobile secretary cloud application replaces aninstructed activity with a self-improved activity. When the nextappointment email related to picking up the kids comes into the emailinbox, the mobile secretary application automatically adds an alarm 2hours before the pickup time to the calendar entry. The mobile secretarycloud application adds an alert to the simple “pick up kids” entry, 2hours before the pick-up time, providing a sound, vibration and/orscreen alert to the user 2 hours before the kids' pick-up time.

In phase 720 the mobile secretary cloud application records whether theuser accepts the self-improved activity, corrects the self-improvedactivity or rejects the self-improved activity with his own activity. Insome embodiments the mobile secretary cloud application may ask the userto suggest what the improvement should be: For example, with a multiplechoice, or free form text entry, or by requesting the user to providevoice feedback which is then voice recognized. For example, Siri on anApple handset and Google Assistant on an Android handset could be usedto implement this functionality. The feedback is then transmitted to thecloud servers for further analysis.

In phase 722 the mobile secretary cloud application records an acceptedself-improved activity as a default option for future use. If the usersimply starts to use the 2-hour alerts, it is likely that theself-improved activity by the mobile secretary cloud application isdesirable by the user as well.

In phase 724 the mobile secretary cloud application records thecorrected version of the corrected self-improved activity as a defaultoption for future use. If it is detected that the user tends to move thealert closer to the activity, for example to 1 hour before the pick-uptime of kids, the mobile secretary cloud application will update thedefault calendar entry to include an alert 1 h before the actual pick-uptime of kids.

In phase 726 the mobile secretary application records the rejectedself-improved activity as failed and does not suggest it again. Thishappens when the user routinely deletes all alerts, and clearly does notwant the mobile phone to alert him.

In one embodiment of the invention the mobile secretary applicationobserves for example from GPS data whether the user arrives in time forthe pick-up of kids. The mobile secretary cloud application can deducefrom email etc. the location of the kids' school, or the user may simplyprovide the location manually to the mobile secretary cloud application.Also, the mobile phones of the children might be connected to the cloudaccount of the user, thereby providing the location of the kids to themobile cloud secretary application. As both the parent and the childlocations are known, the mobile cloud secretary application can be awareof the time and distance separating the parent and the children, andcalculate the time of the alarm for the kid's pick-up accordingly. Inone embodiment the mobile secretary cloud application times the alarm sothat there is a sufficient estimated travel time between the alert andthe pick-up to get to the pick-up location.

In another embodiment, if the estimation of the suitable alert timeproves difficult for the mobile secretary cloud application, in that theresponses of the user indicate dissatisfaction by the user deleting thealerts or changing them multiple times and/or randomly. Then the mobilesecretary application may escalate the issue to the user, and requestthe user to manually specify a suitable alert time for this type ofcalendar entry.

Any features of embodiment 71 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,62, 63, 81, and/or 82 in accordance with the invention. The machinelearning method of embodiment 71 is applicable also to the other earliermentioned use scenarios, such as sorting emails, making reservations,and finding the right route, for example.

FIG. 8A shows an embodiment 81 wherein the machine learning modules areseparate from the mobile secretary cloud application 803C and 803S,wherein “C” denotes the client side and “S” denotes the server side. Inthis embodiment the secretarial functions are implemented separately bythe mobile secretary cloud application 803C and 803S, and it uses thedata and services from a separate cognitive module 804C and 804S,self-improvement module 805C and 805S, and crowd-sourcing module 806Cand 806S. This architecture is useful for example in the situation wherethe operating system already has a cognitive module 804C, 804S,self-improvement module 805C, 805S and/or a crowd-sourcing module 806C,806S.

Cloud services, such as for example Apple Location Services are known totrack iPhone user behavior so that a lot of crowd-sourced data iscollected from probably millions of iPhone and iCloud users. Similarly,Google tracks crowd sourced data of all or some Android users. It is inaccordance with the invention to use this crowd-sourced data with thecognitive module 804C and 804S and/or the self-improvement module 805Cand 805S of the user, with a crowd sourcing software module 806C, 806S,or without.

In one embodiment the cognitive 804C, 804S and the self-improvementmodules 805C, 805S are realized as integrated into the mobile secretarycloud application 803C and 803S, but the crowd-sourced data is obtainedfrom Apple or Google via their respective API's (Application ProgrammerInterface) to avoid duplication of work and activity. Quite clearly, anypermutation or combination of the cognitive 804C, 804S, self-improvement805C, 805S, and crowd-sourcing 806C, 806S-modules is in accordance withthe invention. These modules 804C, 804S, 805C, 805S, 806C, 806S arepreferably implemented with the cloud based client-server architecture.However in some embodiments it is possible that a module is implementedexclusively on the client device only or the server computer only. Forexample the cognitive module 804C might be limited to the client devicein some embodiments, thereby providing better information security andprivacy from the perspective of the individual user.

Any features of embodiment 81 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,62, 63, 71, and/or 82 in accordance with the invention.

FIG. 8B shows an embodiment 82 where the machine learning modules areintegrated into the mobile secretary cloud application. Both the client(C) and the server (S) side software of the cognitive module 804C and804S, self-improvement module 805C and 805S, and crowd-sourcing module806C and 806S are realized within the mobile secretary cloud application803C and 803S. This architecture is useful in that the mobile secretarycloud application is typically more independent from the underlyingoperating system in this way.

Any features of embodiment 82 may be readily combined or permuted withany of the other embodiments 10, 11, 20, 21, 30, 31, 40, 41, 50, 60, 61,62, 63, 71, and/or 81 in accordance with the invention.

The invention has been explained in the aforementioned and sizeableadvantages of the invention have been demonstrated. The invention allowsthe mobile secretary cloud application 602, 612, 622, 632 to replicatethe secretarial functions to a normal consumer at a very small cost. Themobile secretary cloud application 602, 612, 622, 632 produces draftresponses to emails and messages, interacts on the phone on behalf ofthe user, and makes purchases such as flight tickets on behalf of theuser. The mobile secretary cloud application 602, 612, 622, 632 alsosorts and archives emails and messages to different folders, arrangestasks for the user based on priority and urgency, and selects routes andagendas for the day. Similarly, the mobile secretary cloud application602, 612, 622, 632 updates data from one application to the other, forexample emailed agreements of meetings to the calendar and providesdifferent indication sounds alerting the user to the category or vibe ofan incoming message. Any secretarial task can be performed by the mobilesecretary cloud application 602, 612, 622, 632 and cloud network 216,516 as long as the system is trained to produce and distinguish anacceptable secretarial work product with training and validations setsof the said secretarial work product in accordance with the invention.

The invention has been explained above with reference to theaforementioned embodiments. However, it is clear that the invention innot only restricted to these embodiments, but comprises all possibleembodiments within the spirit and scope of the inventive thought and thefollowing patent claims.

REFERENCES

-   US 2013/0110519 A1 DETERMINING USER INTENT BASED ON ONTOLOGIES OF    DOMAINS, published on May 2, 2013, Adam et al.-   WO 2014/197635 A2 INTELLIGENT AUTOMATED ASSISTANT, published on Dec.    11, 2014, Thomas et al.-   US 2016/0155442 A1 EXTENDING DIGITAL PERSONAL ASSISTANT ACTION    PROVIDERS, published on Jun. 2, 2016, Vishvac et al.-   A PRACTICAL INTRODUCTION TO DEEP LEARNING WITH CAFFE AND PYTHON,    published on Jun. 26, 2016, Adil Moujahid.-   INTRODUCING ARTIFICIAL INTELLIGENCE A GRAPHIC GUIDE, Henry Brighton,    Iconbooks 2003.-   MACHINE LEARNING THE NEW AI, Ethem Alpaydin, The MIT Press 2016.-   THE TECHNOLOGICAL SINGULARITY, Murray Shanahan, The MIT Press, 2015.-   THE UNREASONABLE EFFECTIVENESS OF DATA, A. Halevy, P. Norvig and F.    Pereira, IEEE Intelligent Systems 8-12, 2009

The invention claimed is:
 1. A non-transitory computer readable mediumincluding instructions for performing operations for one or moresoftware program products to operate with a client device connected to acloud server network via a communication network, the operationscomprising: reading, by at least one of the software program products,data from a messaging application; operating, by at least one of thesoftware program products, the messaging application, based on the datafrom the messaging application, wherein operating the messagingapplication includes analyzing and sorting messages; and using, by atleast one of the software program products, one or more artificialintelligence models trained using data collected from a plurality ofusers, to at least one of: (i) operate one or more applications forassisting a user to perform tasks or (ii) use the one or moreapplications independently of the user for performing the tasks onbehalf of the user, wherein the tasks include both: (i) providing one ormore draft answers to queries included in inbound messages,automatically to the user via the applications, comprising: analyzingthe inbound messages, using at least one of the artificial intelligencemodels, based on content of the inbound messages; and (ii) organizing anagenda for the user, comprising: scheduling a unit of time and a placefor one or more activities in the agenda based on available time in auser calendar, using at least one of the artificial intelligence models,wherein the cloud server network repeatedly updates artificialintelligence files for a number of tasks, and makes the artificialintelligence files available to the software program products to operatewith the client device, and wherein both: (i) providing the one or moredraft answers to queries and (ii) organizing the agenda, are furtherbased on using crowd sourced data with at least one of the artificialintelligence models, and wherein the providing the one or more draftanswers and the organizing the agenda are based on different crowdsourced data or are based on the same crowd sourced data.
 2. Thenon-transitory computer readable medium of claim 1, wherein theproviding the one or more draft answers to queries included in inboundmessages, automatically to the user via the applications, furthercomprises: generating text for the one or more draft answers, whereinthe at least one of the software program products has at least one of:(i) machine learned the text to be relevant, or (ii) deduced by the oneor more artificial intelligence models that the text is likely be in theone or more draft answers.
 3. The non-transitory computer readablemedium of claim 1, wherein the agenda is organized repeatedly on atleast a daily, business daily, or weekly basis.
 4. The non-transitorycomputer readable medium of claim 1, wherein the one or more draftanswers are provided automatically in response to the inbound messages.5. The non-transitory computer readable medium of claim 1, wherein thesoftware program product identifies a corresponding matter from anemail, historical data of the user, and a calendar entry.
 6. Thenon-transitory computer readable medium of claim 1, wherein the softwareprogram product derives a meaning from an analysis of special charactersin an email.
 7. The non-transitory computer readable medium of claim 1,wherein the software program product forecasts a travel time for atleast one of a calendar or agenda event.
 8. The non-transitory computerreadable medium of claim 1, wherein a location of an appointment is ameasured location of another mobile device of another appointmentparticipant.
 9. The non-transitory computer readable medium of claim 1,wherein the software program product replaces an instructed activitywith a self-improved activity based on improvements recorded in crowdsourced data.
 10. The non-transitory computer readable medium of claim1, wherein the inbound messages comprises an email, and whereinanalyzing the inbound messages comprises associating a meaning to theemail based on semantic content in the email using at least one of theartificial intelligence models.
 11. The non-transitory computer readablemedium of claim 10, wherein the email is segregated into a folder basedon its semantic meaning using the at least one of the artificialintelligence models.
 12. The non-transitory computer readable medium ofclaim 10, wherein all of: (i) contents of messages, (ii) details of oneor more senders, and (iii) date and time of receipt of messages isanalyzed using the at least one of the artificial intelligence models.13. The non-transitory computer readable medium of claim 10, whereinspoken instructions received via Interactive Voice Response System areanalyzed using an artificial intelligence model.
 14. The non-transitorycomputer readable medium of claim 1, wherein at least one of: (i)providing the one or more draft answers to queries or (ii) organizingthe agenda, is further based on a prior input by the user for at leastone of the one or more applications.
 15. The non-transitory computerreadable medium of claim 14, wherein the at least one of: (i) providingthe one or more draft answers to queries or (ii) organizing the agenda,at least partially replicates the prior input by the user using at leastone of the artificial intelligence models.
 16. The non-transitorycomputer readable medium of claim 1, wherein at least one of thesoftware program products analyzes messages from the messagingapplication and selects an indication sound automatically based on oneor more of: contents of the messages, sender of the messages or one ormore recipients of the messages.
 17. The non-transitory computerreadable medium of claim 1, wherein at least one of the software programproducts provides a route calculator via a map application that providesa combined route of driving and walking based on data from all of: (i)the messaging application, (ii) a search application, and (iii) anInternet browser.
 18. The non-transitory computer readable medium ofclaim 1, wherein the one or more applications are operated with usingthe one or more artificial intelligence models to produce the agenda.19. The non-transitory computer readable medium of claim 1, wherein theagenda comprises a plurality of activities and organizing the agenda forthe user comprises determining an optimum total fit of all of thecombined plurality of activities in the agenda for a unit of time. 20.The non-transitory computer readable medium of claim 1, whereinproviding the one or more draft answers to queries included in inboundmessages comprises individually determining draft responses to thequeries, and guiding the user through the responses.
 21. A system forperforming operations for one or more software program products tooperate with a client device connected to a cloud server network via acommunication network, comprising: a processor; and a memory connectedto the processor and storing instructions for the one or more softwareprogram products, wherein the processor is configured to execute thesoftware program products to perform the operations, comprising:reading, by at least one of the software program products, data from amessaging application; operating, by at least one of the softwareprogram products, the messaging application, based on the data from themessaging application, wherein operating the messaging applicationincludes analyzing and sorting messages; and using, by at least one ofthe software program products, one or more artificial intelligencemodels trained using data collected from a plurality of users, to atleast one of: (i) operate one or more applications for assisting a userto perform tasks or (ii) use the one or more applications independentlyof the user for performing the tasks on behalf of the user, wherein thetasks include both: (i) providing one or more draft answers to queriesincluded in inbound messages, automatically to the user via theapplications, comprising: analyzing the inbound messages, using at leastone of the artificial intelligence models, based on content of theinbound messages; and (ii) organizing an agenda for the user,comprising: scheduling a unit of time and a place for one or moreactivities in the agenda based on available time in a user calendar,using at least one of the artificial intelligence models, wherein thecloud server network repeatedly updates artificial intelligence filesfor a number of tasks, and makes the artificial intelligence filesavailable to the software program products to operate with the clientdevice, and wherein both: (i) providing the one or more draft answers toqueries and (ii) organizing the agenda, are further based on using crowdsourced data with at least one of the artificial intelligence models,and wherein the providing the one or more draft answers and theorganizing the agenda are based on different crowd sourced data or arebased on the same crowd sourced data.
 22. The system of claim 21,wherein the providing the one or more draft answers to queries includedin inbound messages, automatically to the user via the applications,further comprises: generating text for the one or more draft answers,wherein the at least one of the software program products has at leastone of: (i) machine learned the text to be relevant, or (ii) deduced bythe one or more artificial intelligence models that the text is likelybe in the one or more draft answers.
 23. The system of claim 21, whereinthe agenda is organized repeatedly on at least a daily, business daily,or weekly basis.
 24. The system of claim 21, wherein the one or moredraft answers are provided automatically in response to the inboundmessages.
 25. The system of claim 21, wherein the software programproduct identifies a corresponding matter from an email, historical dataof the user, and a calendar entry.
 26. The system of claim 21, whereinthe software program product derives a meaning from an analysis ofspecial characters in an email.
 27. The system of claim 21, wherein thesoftware program product forecasts a travel time for at least one of acalendar or agenda event.
 28. The system of claim 21, wherein a locationof an appointment is a measured location of another mobile device ofanother appointment participant.
 29. The system of claim 21, wherein thesoftware program product replaces an instructed activity with aself-improved activity based on improvements recorded in crowd sourceddata.
 30. The system of claim 21, wherein the inbound messages comprisesan email, and wherein analyzing the inbound messages comprisesassociating a meaning to the email based on semantic content in theemail using at least one of the artificial intelligence models.
 31. Thesystem of claim 30, wherein the email is segregated into a folder basedon its semantic meaning using the at least one of the artificialintelligence models.
 32. The system of claim 30, wherein all of: (i)contents of messages, (ii) details of one or more senders, and (iii)date and time of receipt of messages is analyzed using the at least oneof the artificial intelligence models.
 33. The system of claim 30,wherein spoken instructions received via Interactive Voice ResponseSystem are analyzed using an artificial intelligence model.
 34. Thesystem of claim 21, wherein at least one of: (i) providing the one ormore draft answers to queries or (ii) organizing the agenda, is furtherbased on a prior input by the user for at least one of the one or moreapplications.
 35. The system of claim 34, wherein the at least one of:(i) providing the one or more draft answers to queries or (ii)organizing the agenda, at least partially replicates the prior input bythe user using at least one of the artificial intelligence models. 36.The system of claim 21, wherein at least one of the software programproducts analyzes messages from the messaging application and selects anindication sound automatically based on one or more of: contents of themessages, sender of the messages or one or more recipients of themessages.
 37. The system of claim 21, wherein at least one of thesoftware program products provides a route calculator via a mapapplication that provides a combined route of driving and walking basedon data from all of: (i) the messaging application, (ii) a searchapplication, and (iii) an Internet browser.
 38. The system of claim 21,wherein the one or more applications are operated with using the one ormore artificial intelligence models to produce the agenda.
 39. Thesystem of claim 21, wherein the agenda comprises a plurality ofactivities and organizing the agenda for the user comprises determiningan optimum total fit of all of the combined plurality of activities inthe agenda for a unit of time.
 40. The system of claim 21, whereinproviding the one or more draft answers to queries included in inboundmessages comprises individually determining draft responses to thequeries, and guiding the user through the responses.